- name: Build ARM Linux py3.6-9 wheels
env:
CIBW_BUILD: "cp36-* cp37-* cp38-* cp39-*"
- - name: Build ARM Linux py3.10-12 wheels
+ - name: Build ARM Linux py3.10-13 wheels
env:
- CIBW_BUILD: "cp310-* cp311-* cp312-*"
+ CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-*"
- name: Build ARM macOS wheels
macos_instance:
image: ghcr.io/cirruslabs/macos-sonoma-base:latest
env:
- CIBW_BUILD: "cp39-* cp310-* cp311-* cp312-*"
+ CIBW_BUILD: "cp39-* cp310-* cp311-* cp312-* cp313-*"
alias: build_wheels
CIBW_SKIP: "*-musllinux_*"
CIBW_BUILD_VERBOSITY: 1
+ # Avoid linking with non-system library libdeflate.dylib
+ CIBW_ENVIRONMENT_MACOS: HTSLIB_CONFIGURE_OPTIONS="--without-libdeflate"
+
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28
install_script: |
python3 -m venv $VENV
- pip3 install cibuildwheel==2.17.0
+ pip3 install cibuildwheel==2.22.0
build_script: |
cibuildwheel
-Many people have contributed to pysam. The list of github contributors
+Many people have contributed to pysam. The list of GitHub contributors
is the best place to get a full list of authors and their contributions.
The list and summary below is a out-of-date and represents the earlier
stages of the project.
List of contributors:
-Andreas Heger, Tildon Grant Belgard, Florian Finkernagel, Leo
-Goodstadt, Martin Goodson all contributed code to pysam.
+Andreas Heger, Tildon Grant Belgard, Florian Finkernagel,
+Leo Goodstadt, Martin Goodson all contributed code to pysam.
John Marshall has been looking after pysam and its community for
several years, as well as making many code contributions and improving
Gerton Lunter provided a validating VCF parser.
-Marcel Martin implemented python 3 compatibility.
+Marcel Martin implemented Python 3 compatibility and added type hints.
Ben Schiller contributed a Windows compatible clone.
The sources in the directory samtools are from the samtools project:
SAM/BAM specification. He designed and implemented the BGZF format, the
underlying indexable compression format for the BAM format. BGZF does
not support arithmetic between file offsets.
-
-Jue Ruan for the Beijing Genome Institute designed and implemented the
-RAZF format, an alternative indexable compression format. RAZF supports
-arithmetic between file offsets, at the cost of increased index file
-size and the full compatibility with gzip. RAZF is optional and only
-used in `faidx' for indexing RAZF compressed fasta files.
-
-Colin Hercus updated novo2sam.pl to support gapped alignment by
-novoalign.
-
Installing pysam
================
-Pysam can be installed through conda_, pypi_ and from the repository.
+Pysam can be installed through conda_, PyPI_ and from the repository.
The recommended way to install pysam is through conda/bioconda.
Conda installation
will be set automatically, which will potentially save a lot of
trouble on OS X.
-Pypi installation
+PyPI installation
=================
Pysam provides a python interface to the functionality contained
Builtin
-------
-The typical installation will be through pypi_::
+The typical installation will be through PyPI_::
pip install pysam
python setup.py install
-For compilation options, see the section on Pypi installation above.
+For compilation options, see the section on PyPI installation above.
Requirements
============
include win32/*.[ch]
-# exclude tests from pypi tar-ball - they
-# require additional data
+# exclude tests from sdist tarball as they require additional data
prune tests/
# samtools
exclude htslib/*config*.h
include htslib/configure.ac htslib/m4/*.m4 htslib/*.in
-include htslib/configure htslib/version.sh
+include htslib/configure htslib/config.guess htslib/config.sub htslib/version.sh
include htslib/Makefile htslib/*.mk
exclude htslib/config.mk htslib/htscodecs.mk
.. An online version of the release history can be found here:
.. http://pysam.readthedocs.io/en/latest/release.html
+Release 0.23.0
+==============
+
+.. rubric:: 5 February 2025
+
+This pysam release wraps htslib/samtools/bcftools 1.21 (PR #1310).
+
+It has been tested with Python versions 3.6 through 3.13, and wheels are
+available via PyPI_ for all of those Python versions. Python versions 3.6
+through 3.8 are end-of-life; particularly if you use pysam with one of
+these versions, please vote in the version survey at issue #1230.
+
+This is expected to be the last release supporting Python 3.6 and 3.7.
+
+Bugs fixed:
+
+* Fix :func:`pysam.samtools.command(save_stdout=filename) <.samtools.command>`
+ redirection of output to a file. (#677, reported by Haowen Zhang and analysed
+ by Youri Hoogstrate)
+
+* Reinstate HTTPS/S3/GCS support in pre-built Linux wheels when used on Debian
+ and Ubuntu: these wheels now work around the discrepancy between Red Hat and
+ Debian CA Certificate file locations. (#1257, #1268, reported by Daisie Huang
+ and Li Tai Fang)
+
+* Calling :meth:`.VariantHeader.new_record` repeatedly with the same
+ :obj:`samples` object now sets GT correctly every time. (#1308, reported
+ by Arthur Gymer)
+
+* Correct the exception produced when :meth:`.AlignedSegment.set_tags` is
+ used with an invalid value type. (#1233, PR #1235, reported by Weisheng Wu
+ and Marcus Stoiber)
+
+* Many type hinting corrections. (#1298, #1316, PR #1296, PR #1306, PR #1313,
+ PR #1315, thanks to Victor Epain, Arthur Gymer, @mshunjan, and Matt Stone)
+
+* The undocumented :func:`!pysam.samtools.import_` alias for invoking
+ ``samtools import`` has been removed;
+ use :func:`pysam.samtools.fqimport() <.samtools.command>` instead.
+
+* Corrections to several test data files to account for HTSlib 1.20 and later's
+ improved validity checking. (#1291, reported by David Seifert; etc)
+
+New functionality:
+
+* :meth:`.AlignedSegment.get_aligned_pairs` now optionally returns the
+ associated CIGAR operator in each position tuple. (#1292, PR #1294,
+ thanks to Lara Fuhrmann and Ivan Blagoev Topolsky)
+
+* New :meth:`AlignmentFile.flush() <.HTSFile.flush>` and
+ :meth:`VariantFile.flush() <.HTSFile.flush>` methods for flushing buffered
+ output to streams. (#1299, requested by @blex-max)
+
+* Improved :class:`str() <str>` for :class:`.AlignedSegment` now displays
+ reference sequence names when they are available and uses ``#N`` only when
+ necessary. (#1318, requested by Liu)
+
+* Implement :func:`repr` for :class:`.AlignedSegment` so that it displays the
+ most useful fields. (PR #1267, thanks to Marcel Martin)
+
+* Pysam's tests can now run in parallel. (#1284, reported by Yuri Victorovich)
+
+Documentation improvements:
+
+* More complete documentation of invoking SAMtools and BCFtools subcommands
+ via Pysam. (#1096, #1241, PR #1261, PR #1275, #1323, et al, reported by
+ Seung-been "Steven" Lee, Robert Baldwin, Michael Hall, Indraniel Das,
+ and @shokrofont)
+
+* Use the Pysam names for CIGAR operators rather than internal C names.
+ (#1255, reported by Ilya Shlyakhter)
+
+* Mention :meth:`AlignmentFile.fetch("*") <.AlignmentFile.fetch>` in the
+ FAQ entry on fetching unmapped reads. (#424, reported by Ben Weisburd)
+
+
Release 0.22.1
==============
Bugs fixed:
* Preserve all header field tags defined in the SAM specification (notably TP)
- in :meth:`.AlignmentHeader.from_dict` and :meth:`.AlignmentHeader.to_dict`
+ in :meth:`.AlignmentHeader.from_dict` and :meth:`.AlignmentHeader.to_dict`.
(#1237, PR #1238, thanks to Tim Fennell and Nils Homer)
* Adjust HTSlib's Makefile so that ``make distclean`` no longer tries to
- rebuild the htscodecs configury (PR #1247, reported by Nicola Soranzo)
+ rebuild the htscodecs configury. (PR #1247, reported by Nicola Soranzo)
* Reinstate S3 support in pre-built Linux wheels: support for this protocol
- was inadvertently omitted from the pre-built 0.22.0 wheels on Linux
+ was inadvertently omitted from the pre-built 0.22.0 wheels on Linux.
(#1249, #1277, etc varying circumstances; likely it is this that was
reported by Mathew Baines, Benjamin Sargsyan, et al)
-* Add missing :attr:`.AlignedSegment.is_mapped` etc properties to type stubs
+* Add missing :attr:`.AlignedSegment.is_mapped` etc properties to type stubs.
(PR #1273, thanks to Matt Stone)
-* Fix off-by-one NamedTupleProxy, :class:`.asBed`, etc array bounds check
+* Fix off-by-one NamedTupleProxy, :class:`.asBed`, etc array bounds check.
(#1279, reported by Dan Bolser)
-* Make pysam's klib headers compatible with C++ (reported by Martin Grigorov)
+* Make pysam's klib headers compatible with C++. (reported by Martin Grigorov)
Release 0.22.0
This pysam release wraps htslib/samtools/bcftools 1.18 (PR #1208).
It has been tested with Python versions 3.6 through 3.12, and wheels are
-available via pypi_ for all of those Python versions. Python versions 3.6
+available via PyPI_ for all of those Python versions. Python versions 3.6
and 3.7 are end-of-life; particularly if you use pysam with either of
these versions, please vote in the version survey at issue #1230.
Bugs fixed:
-* Remove Cython from runtime dependencies (PR #1186, thanks to Nicola Soranzo,
+* Remove Cython from runtime dependencies. (PR #1186, thanks to Nicola Soranzo,
also reported by Arya Massarat in PR #1194)
-* Miscellaneous dependency improvements (PR #1216, #1217, PR #1218, PR #1219,
+* Miscellaneous dependency improvements. (PR #1216, #1217, PR #1218, PR #1219,
thanks to Martin Larralde and Arthur Vigil)
* Suppress spurious "Could not retrieve index file" message when opening an
- AlignmentFile (#939, #1214, reported by ChengYong Tham and Sebastian Röner)
+ AlignmentFile. (#939, #1214, reported by ChengYong Tham and Sebastian Röner)
-* Propagate SAM parsing errors encounted in :meth:`.AlignedSegment.fromstring`
+* Propagate SAM parsing errors encounted in :meth:`.AlignedSegment.fromstring`.
(#1196, reported by DV Klopfenstein)
* Accept invalid MD:A tagged fields produced by HTSeq instead of crashing
in :meth:`AlignedSegment.get_aligned_pairs(with_seq=True)
- <.AlignedSegment.get_aligned_pairs>` (#1226, reported by Isaac Vock)
+ <.AlignedSegment.get_aligned_pairs>`. (#1226, reported by Isaac Vock)
-* Fix multiarch macOS CI builds by removing brewed liblzma (#1205, reported
+* Fix multiarch macOS CI builds by removing brewed liblzma. (#1205, reported
by Till Hartmann)
-* Fix :attr:`.VariantRecordSample.alleles` type hint (#1179, reported by
+* Fix :attr:`.VariantRecordSample.alleles` type hint. (#1179, reported by
David Seifert)
New functionality:
* Add optional :meth:`HTSFile.seek(..., whence) <.HTSFile.seek>` parameter
- and clarify which functions use libc.SEEK_SET vs io.SEEK_SET
+ and clarify which functions use libc.SEEK_SET vs io.SEEK_SET.
(#1185, requested by luyulin)
-* File handling improvements in samtools & bcftools commands (should improve
+* File handling improvements in samtools & bcftools commands. (should improve
#1193 and #1195, reported by Rob Bierman and Sam Chorlton)
-* Improve :class:`.FastxFile` performance (PR #1227, thanks to Fabian Klötzl
+* Improve :class:`.FastxFile` performance. (PR #1227, thanks to Fabian Klötzl
and Valentyn Bezshapkin)
-* Improve the accuracy of type hints for :class:`.AlignmentFile` iteration
+* Improve the accuracy of type hints for :class:`.AlignmentFile` iteration.
(#1184, PR #1189, reported by @PikalaxALT)
Documentation improvements:
-* Clarify that :meth:`.AlignedSegment.get_aligned_pairs` results are 0-based
+* Clarify that :meth:`.AlignedSegment.get_aligned_pairs` results are 0-based.
(#1180, reported by Nick Semenkovich)
-* Clarify :meth:`.AlignedSegment.get_reference_positions` documentation
+* Clarify :meth:`.AlignedSegment.get_reference_positions` documentation.
(#836, #838, reported by Liang Ou and Nick Stoler)
* Clarify that installation via pip usually uses a wheel, and that configuring
the build via $HTSLIB_CONFIGURE_OPTIONS etc only applies when installing from
- an sdist (#1086, reported by Layne Sadler)
+ an sdist. (#1086, reported by Layne Sadler)
A message from pysam's founder, Andreas Heger:
compilation options. Especially for OS X this will potentially save a
lot of trouble.
-The current version of pysam wraps 3rd-party code from htslib-1.18, samtools-1.18, and bcftools-1.18.
+The current version of pysam wraps 3rd-party code from htslib-1.21, samtools-1.21, and bcftools-1.21.
-Pysam is available through `pypi
-<https://pypi.python.org/pypi/pysam>`_. To install, type::
+Pysam is available through `PyPI <https://pypi.org/project/pysam/>`_.
+To install, type::
pip install pysam
* @sites: list of positions
*
* Same as hmm_run_fwd_bwd, in addition a pointer to a matrix with the new
- * transition probabilities is returned. In this verison, emission
+ * transition probabilities is returned. In this version, emission
* probabilities are not updated.
*/
double *hmm_run_baum_welch(hmm_t *hmm, int nsites, double *eprob, uint32_t *sites);
The MIT/Expat License
-Copyright (C) 2012-2023 Genome Research Ltd.
+Copyright (C) 2012-2024 Genome Research Ltd.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+-----------------------------------------------------------------------------
+
+License for edlib.[ch]
+
+The MIT License (MIT)
+
+Copyright (c) 2014 Martin Šošić
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
/* The MIT License
- Copyright (c) 2021-2023 Genome Research Ltd.
+ Copyright (c) 2021-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
kstring_t ref, alt;
int ial; // the index of the original ALT allele, 1-based
int beg, end; // 0-based inclusive offsets to ref,alt
+ int plen; // the ref,alt prefix length, eg plen=1 for C>CA
}
atom_t;
atom->alt.l = 0;
kputc(refb, &atom->ref);
kputc(refb, &atom->alt);
- atom->beg = atom->end = i;
- atom->ial = ial;
+ atom->beg = atom->end = i;
+ atom->ial = ial;
+ atom->plen = 1;
}
continue;
}
if ( rcmp ) return rcmp;
return strcasecmp(a->alt.s,b->alt.s);
}
+
+// returns
+// 0 .. identical beg,ref,alt
+// 1 .. non-overlapping variants, but record may overlap (A>AT vs A>C)
+// 2 .. overlapping (conflicting) variants
+static int _atoms_overlap(const atom_t *a, const atom_t *b)
+{
+ if ( a->beg < b->beg ) return 2;
+ if ( a->beg > b->beg ) return 2;
+
+ // consider SNV followed by DEL as not overlapping
+ // CC > C a.plen=1 (ref,alt prefix len=1)
+ // C > T b.plen=0 (ref,alt prefix len=0)
+ if ( a->plen && a->plen >= b->ref.l ) return 1;
+ if ( b->plen && b->plen >= a->ref.l ) return 1;
+
+ int rcmp = strcasecmp(a->ref.s,b->ref.s);
+ if ( rcmp ) return 2;
+
+ // consider SNV followed by INS as not overlapping
+ // A > AT a.plen=1 (ref,alt prefix len=1)
+ // A > C b.plen=0 (ref,alt prefix len=0)
+ if ( a->plen && a->plen >= b->alt.l ) return 1;
+ if ( b->plen && b->plen >= a->alt.l ) return 1;
+
+ rcmp = strcasecmp(a->alt.s,b->alt.s);
+ if ( rcmp ) return 2;
+ return 0;
+}
/*
For reproducibility of tests on different platforms, we need to guarantee the same order of identical
atoms originating from different source ALTs. Even though they are consistent, different values can be
static void _split_table_overlap(abuf_t *buf, int iout, atom_t *atom)
{
uint8_t *ptr = buf->split.tbl + iout*buf->split.nori;
- ptr[atom->ial-1] = _atoms_inconsistent(atom,buf->split.atoms[iout]) ? 2 : 1;
+ int olap = _atoms_overlap(atom,buf->split.atoms[iout]);
+ ptr[atom->ial-1] = olap > 1 ? 2 : 1;
+
+ // The test test/atomize.split.5.vcf shows why we sometimes can and sometimes
+ // cannot remove the star allele like this
+ // buf->split.overlaps[iout] = olap > 1 ? 1 : 0;
+ // I forgot the details of the code, so don't immediately see
+ // if this could be made smarter
buf->split.overlaps[iout] = 1;
}
#if 0
buf->tmp2 = dst.s;
ret = bcf_update_info(buf->out_hdr, out, tag, buf->tmp2, dst.l, type);
}
- if ( ret!=0 ) error("An error occurred while updating INFO/%s\n",tag);
+ if ( ret!=0 ) error("An error occurred while updating INFO/%s (errcode=%d)\n",tag,ret);
}
}
static void _split_table_set_history(abuf_t *buf)
{
- int i,j;
+ int i,j,ret;
bcf1_t *rec = buf->split.rec;
+
+ // Don't update if the tag already exists. This is to prevent -a from overwriting -m
+ int m = 0;
+ char *tmp = NULL;
+ ret = bcf_get_info_string(buf->hdr,rec,buf->split.info_tag,&tmp,&m);
+ free(tmp);
+ if ( ret>0 ) return;
+
buf->tmps.l = 0;
ksprintf(&buf->tmps,"%s|%"PRIhts_pos"|%s|",bcf_seqname(buf->hdr,rec),rec->pos+1,rec->d.allele[0]);
for (i=1; i<rec->n_allele; i++)
kputc(',',&buf->tmps);
}
buf->tmps.s[--buf->tmps.l] = 0;
- if ( (bcf_update_info_string(buf->out_hdr, out, buf->split.info_tag, buf->tmps.s))!=0 )
- error("An error occurred while updating INFO/%s\n",buf->split.info_tag);
+ if ( (ret=bcf_update_info_string(buf->out_hdr, out, buf->split.info_tag, buf->tmps.s))!=0 )
+ error("An error occurred while updating INFO/%s (errcode=%d)\n",buf->split.info_tag,ret);
}
}
static void _split_table_set_gt(abuf_t *buf)
#undef BRANCH
ret = bcf_update_format(buf->out_hdr, out, tag, buf->tmp2, 3*(1+star_allele)*nsmpl, type);
}
- if ( ret!=0 ) error("An error occurred while updating FORMAT/%s\n",tag);
+ if ( ret!=0 ) error("An error occurred while updating FORMAT/%s (errcode=%d)\n",tag,ret);
}
}
static inline int _is_acgtn(char *seq)
_split_table_init(buf,rec,buf->natoms);
for (i=0; i<buf->natoms; i++)
{
- if ( i && !_atoms_inconsistent(&buf->atoms[i-1],&buf->atoms[i]) ) continue;
+ if ( i && _atoms_inconsistent(&buf->atoms[i-1],&buf->atoms[i])==0 ) continue;
_split_table_new(buf, &buf->atoms[i]); // add a new unique output atom
}
for (i=0; i<buf->natoms; i++)
/* The MIT License
- Copyright (c) 2021-2023 Genome Research Ltd.
+ Copyright (c) 2021-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
kstring_t ref, alt;
int ial; // the index of the original ALT allele, 1-based
int beg, end; // 0-based inclusive offsets to ref,alt
+ int plen; // the ref,alt prefix length, eg plen=1 for C>CA
}
atom_t;
atom->alt.l = 0;
kputc(refb, &atom->ref);
kputc(refb, &atom->alt);
- atom->beg = atom->end = i;
- atom->ial = ial;
+ atom->beg = atom->end = i;
+ atom->ial = ial;
+ atom->plen = 1;
}
continue;
}
if ( rcmp ) return rcmp;
return strcasecmp(a->alt.s,b->alt.s);
}
+
+// returns
+// 0 .. identical beg,ref,alt
+// 1 .. non-overlapping variants, but record may overlap (A>AT vs A>C)
+// 2 .. overlapping (conflicting) variants
+static int _atoms_overlap(const atom_t *a, const atom_t *b)
+{
+ if ( a->beg < b->beg ) return 2;
+ if ( a->beg > b->beg ) return 2;
+
+ // consider SNV followed by DEL as not overlapping
+ // CC > C a.plen=1 (ref,alt prefix len=1)
+ // C > T b.plen=0 (ref,alt prefix len=0)
+ if ( a->plen && a->plen >= b->ref.l ) return 1;
+ if ( b->plen && b->plen >= a->ref.l ) return 1;
+
+ int rcmp = strcasecmp(a->ref.s,b->ref.s);
+ if ( rcmp ) return 2;
+
+ // consider SNV followed by INS as not overlapping
+ // A > AT a.plen=1 (ref,alt prefix len=1)
+ // A > C b.plen=0 (ref,alt prefix len=0)
+ if ( a->plen && a->plen >= b->alt.l ) return 1;
+ if ( b->plen && b->plen >= a->alt.l ) return 1;
+
+ rcmp = strcasecmp(a->alt.s,b->alt.s);
+ if ( rcmp ) return 2;
+ return 0;
+}
/*
For reproducibility of tests on different platforms, we need to guarantee the same order of identical
atoms originating from different source ALTs. Even though they are consistent, different values can be
static void _split_table_overlap(abuf_t *buf, int iout, atom_t *atom)
{
uint8_t *ptr = buf->split.tbl + iout*buf->split.nori;
- ptr[atom->ial-1] = _atoms_inconsistent(atom,buf->split.atoms[iout]) ? 2 : 1;
+ int olap = _atoms_overlap(atom,buf->split.atoms[iout]);
+ ptr[atom->ial-1] = olap > 1 ? 2 : 1;
+
+ // The test test/atomize.split.5.vcf shows why we sometimes can and sometimes
+ // cannot remove the star allele like this
+ // buf->split.overlaps[iout] = olap > 1 ? 1 : 0;
+ // I forgot the details of the code, so don't immediately see
+ // if this could be made smarter
buf->split.overlaps[iout] = 1;
}
#if 0
buf->tmp2 = dst.s;
ret = bcf_update_info(buf->out_hdr, out, tag, buf->tmp2, dst.l, type);
}
- if ( ret!=0 ) error("An error occurred while updating INFO/%s\n",tag);
+ if ( ret!=0 ) error("An error occurred while updating INFO/%s (errcode=%d)\n",tag,ret);
}
}
static void _split_table_set_history(abuf_t *buf)
{
- int i,j;
+ int i,j,ret;
bcf1_t *rec = buf->split.rec;
+
+ // Don't update if the tag already exists. This is to prevent -a from overwriting -m
+ int m = 0;
+ char *tmp = NULL;
+ ret = bcf_get_info_string(buf->hdr,rec,buf->split.info_tag,&tmp,&m);
+ free(tmp);
+ if ( ret>0 ) return;
+
buf->tmps.l = 0;
ksprintf(&buf->tmps,"%s|%"PRIhts_pos"|%s|",bcf_seqname(buf->hdr,rec),rec->pos+1,rec->d.allele[0]);
for (i=1; i<rec->n_allele; i++)
kputc(',',&buf->tmps);
}
buf->tmps.s[--buf->tmps.l] = 0;
- if ( (bcf_update_info_string(buf->out_hdr, out, buf->split.info_tag, buf->tmps.s))!=0 )
- error("An error occurred while updating INFO/%s\n",buf->split.info_tag);
+ if ( (ret=bcf_update_info_string(buf->out_hdr, out, buf->split.info_tag, buf->tmps.s))!=0 )
+ error("An error occurred while updating INFO/%s (errcode=%d)\n",buf->split.info_tag,ret);
}
}
static void _split_table_set_gt(abuf_t *buf)
#undef BRANCH
ret = bcf_update_format(buf->out_hdr, out, tag, buf->tmp2, 3*(1+star_allele)*nsmpl, type);
}
- if ( ret!=0 ) error("An error occurred while updating FORMAT/%s\n",tag);
+ if ( ret!=0 ) error("An error occurred while updating FORMAT/%s (errcode=%d)\n",tag,ret);
}
}
static inline int _is_acgtn(char *seq)
_split_table_init(buf,rec,buf->natoms);
for (i=0; i<buf->natoms; i++)
{
- if ( i && !_atoms_inconsistent(&buf->atoms[i-1],&buf->atoms[i]) ) continue;
+ if ( i && _atoms_inconsistent(&buf->atoms[i-1],&buf->atoms[i])==0 ) continue;
_split_table_new(buf, &buf->atoms[i]); // add a new unique output atom
}
for (i=0; i<buf->natoms; i++)
/* bam2bcf.c -- variant calling.
Copyright (C) 2010-2012 Broad Institute.
- Copyright (C) 2012-2022 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk>
{
int i, n, ref4, is_indel, ori_depth = 0;
+#ifdef GLF_DEBUG
+ fprintf(stderr, "Call GLFGEN\n");
+#endif
+
// clean from previous run
r->ori_depth = 0;
r->mq0 = 0;
bca->bases = (uint16_t*)realloc(bca->bases, 2 * bca->max_bases);
}
+ // Detect if indel occurs anywhere in this sample
+ int indel_in_sample = 0;
+ if (bca->edlib) {
+ for (i = n = 0; i < _n; ++i) {
+ const bam_pileup1_t *p = pl + i;
+ if (p->indel) indel_in_sample = 1;
+ }
+ }
+
// fill the bases array
double nqual_over_60 = bca->nqual / 60.0;
int ADR_ref_missed[4] = {0};
b = p->aux>>16&0x3f; // indel type
seqQ = q = (p->aux & 0xff); // mp2 + builtin indel-bias
- if ( !bca->indels_v20 )
+ if (bca->edlib) {
+ if (indel_in_sample) {
+ seqQ = q = (p->aux & 0xff); // mp2 + builtin indel-bias
+ } else if (p->aux & 0xff) {
+ // An indel in another sample, but not this. So just use
+ // basic sequence confidences.
+ q = bam_get_qual(p->b)[p->qpos];
+ if (q > bca->max_baseQ) q = bca->max_baseQ;
+ seqQ = 99;
+ }
+ }
+
+ if ( !bca->indels_v20 && !bca->edlib )
{
/*
This heuristics was introduced by e4e161068 and claims to fix #1446. However, we obtain
}
}
+#ifdef GLF_DEBUG
+ fprintf(stderr, "GLF %s\t%d\t%d\n", bam_get_qname(p->b),
+ bca->indel_types[b], q);
+#endif
if (q < bca->min_baseQ)
{
if (!p->indel && b < 4) // not an indel read
}
continue;
}
+
+#ifndef MIN
+#define MIN(a,b) ((a)<(b)?(a):(b))
+#endif
+
+#ifndef MAX
+#define MAX(a,b) ((a)>(b)?(a):(b))
+#endif
+
+#if 1 // TEST 6
+ if (bca->edlib) {
+ // Deeper data should rely more heavily on counts of data
+ // than quality, as quality can be unreliable and prone to
+ // miscalculations through BAQ, STR analysis, etc.
+ // So we put a cap on how good seqQ can be.
+ //
+ // Is it simply the equivalent of increasing -F filter?
+ // Not quite, as the latter removes many real variants upfront.
+ // This calls them and then post-adjusts quality, potentially
+ // dropping it later or changing genotype. So we still get
+ // calls, but lower qual.
+ seqQ = MIN(seqQ, bca->seqQ_offset-(MIN(20,_n)*5));
+
+ if (indel_in_sample && p->indel == 0 && b != 0) {
+ // This read doesn't contain an indel in CIGAR, but it
+ // is assigned to an indel now (b != 0), These are
+ // reads we've corrected with realignment, but they're
+ // also enriched for FPs so at high depth we reduce their
+ // confidence and let the depth do the talking. If it's
+ // real and deep, then we don't need every read aligning.
+ // We also reduce base quality too to reflect the
+ // chance of our realignment being incorrect.
+
+ seqQ = MIN(seqQ, seqQ/2 + 5); // q2p5
+
+ // Finally reduce indel quality.
+ // This is a blend of indelQ and base QUAL.
+ q = MIN((int)bam_get_qual(p->b)[p->qpos]/4+10, q/4+1);
+ }
+ }
+#endif
+
+ // Note baseQ changes some output fields such as I16, but has no
+ // significant affect on "call".
baseQ = p->aux>>8&0xff;
}
else
}
mapQ = p->b->core.qual < 255? p->b->core.qual : DEF_MAPQ; // special case for mapQ==255
if ( !mapQ ) r->mq0++;
+#ifdef GLF_DEBUG
+ fprintf(stderr, "GLF2 %s\t%d\t%d\t%d,%d\n",
+ bam_get_qname(p->b), b, q,
+ seqQ, mapQ);
+#endif
if (q > seqQ) q = seqQ;
mapQ = mapQ < bca->capQ? mapQ : bca->capQ;
if (q > mapQ) q = mapQ;
for (i=0; i<4; i++) r->ADF[i] += lroundf((float)dp_ambig * r->ADF[i]/dp);
}
+ // Else consider downgrading bca->bases[] scores by AD vs AD_ref_missed
+ // ratios. This is detrimental on Illumina, but beneficial on PacBio CCS.
+ // It's possibly related to the homopolyer error likelihoods or overall
+ // Indel accuracy. Maybe tie this in to the -h option?
+
r->ori_depth = ori_depth;
// glfgen
errmod_cal(bca->e, n, 5, bca->bases, r->p); // calculate PL of each genotype
+
+ // TODO: account for the number of unassigned reads. If depth is 50,
+ // but AD is 5,7 then it may look like a variant but it probably
+ // should be low quality.
+
return n;
}
if ( bc->ori_ref < 0 )
{
bcf_update_info_flag(hdr, rec, "INDEL", NULL, 1);
- if ( fmt_flag&B2B_INFO_IDV )
- bcf_update_info_int32(hdr, rec, "IDV", &bca->max_support, 1);
- if ( fmt_flag&B2B_INFO_IMF )
- bcf_update_info_float(hdr, rec, "IMF", &bca->max_frac, 1);
+ uint32_t idv = bca->max_support;
+ if ( fmt_flag&B2B_INFO_IMF) {
+ float max_frac;
+ // Recompute IDV and IMF based on alignment results for more
+ // accurate counts, but only when in new "--indels-cns" mode.
+ if (bc->ADF && bc->ADR && bca->edlib) {
+ int max_ad = 0;
+ for (int k = 1; k < rec->n_allele; k++) {
+ if (max_ad < bc->ADF[k] + bc->ADR[k])
+ max_ad = bc->ADF[k] + bc->ADR[k];
+ }
+ max_frac = (double)(max_ad) / bc->ori_depth;
+ idv = max_ad;
+ } else {
+ max_frac = bca->max_frac;
+ }
+ // Copied here to maintain order for consistency of "make check"
+ if ( fmt_flag&B2B_INFO_IDV )
+ bcf_update_info_int32(hdr, rec, "IDV", &idv, 1);
+ bcf_update_info_float(hdr, rec, "IMF", &max_frac, 1);
+ } else {
+ if ( fmt_flag&B2B_INFO_IDV )
+ bcf_update_info_int32(hdr, rec, "IDV", &idv, 1);
+ }
}
bcf_update_info_int32(hdr, rec, "DP", &bc->ori_depth, 1);
if ( fmt_flag&B2B_INFO_ADF )
/* bam2bcf.c -- variant calling.
Copyright (C) 2010-2012 Broad Institute.
- Copyright (C) 2012-2022 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk>
{
int i, n, ref4, is_indel, ori_depth = 0;
+#ifdef GLF_DEBUG
+ fprintf(bcftools_stderr, "Call GLFGEN\n");
+#endif
+
// clean from previous run
r->ori_depth = 0;
r->mq0 = 0;
bca->bases = (uint16_t*)realloc(bca->bases, 2 * bca->max_bases);
}
+ // Detect if indel occurs anywhere in this sample
+ int indel_in_sample = 0;
+ if (bca->edlib) {
+ for (i = n = 0; i < _n; ++i) {
+ const bam_pileup1_t *p = pl + i;
+ if (p->indel) indel_in_sample = 1;
+ }
+ }
+
// fill the bases array
double nqual_over_60 = bca->nqual / 60.0;
int ADR_ref_missed[4] = {0};
b = p->aux>>16&0x3f; // indel type
seqQ = q = (p->aux & 0xff); // mp2 + builtin indel-bias
- if ( !bca->indels_v20 )
+ if (bca->edlib) {
+ if (indel_in_sample) {
+ seqQ = q = (p->aux & 0xff); // mp2 + builtin indel-bias
+ } else if (p->aux & 0xff) {
+ // An indel in another sample, but not this. So just use
+ // basic sequence confidences.
+ q = bam_get_qual(p->b)[p->qpos];
+ if (q > bca->max_baseQ) q = bca->max_baseQ;
+ seqQ = 99;
+ }
+ }
+
+ if ( !bca->indels_v20 && !bca->edlib )
{
/*
This heuristics was introduced by e4e161068 and claims to fix #1446. However, we obtain
}
}
+#ifdef GLF_DEBUG
+ fprintf(bcftools_stderr, "GLF %s\t%d\t%d\n", bam_get_qname(p->b),
+ bca->indel_types[b], q);
+#endif
if (q < bca->min_baseQ)
{
if (!p->indel && b < 4) // not an indel read
}
continue;
}
+
+#ifndef MIN
+#define MIN(a,b) ((a)<(b)?(a):(b))
+#endif
+
+#ifndef MAX
+#define MAX(a,b) ((a)>(b)?(a):(b))
+#endif
+
+#if 1 // TEST 6
+ if (bca->edlib) {
+ // Deeper data should rely more heavily on counts of data
+ // than quality, as quality can be unreliable and prone to
+ // miscalculations through BAQ, STR analysis, etc.
+ // So we put a cap on how good seqQ can be.
+ //
+ // Is it simply the equivalent of increasing -F filter?
+ // Not quite, as the latter removes many real variants upfront.
+ // This calls them and then post-adjusts quality, potentially
+ // dropping it later or changing genotype. So we still get
+ // calls, but lower qual.
+ seqQ = MIN(seqQ, bca->seqQ_offset-(MIN(20,_n)*5));
+
+ if (indel_in_sample && p->indel == 0 && b != 0) {
+ // This read doesn't contain an indel in CIGAR, but it
+ // is assigned to an indel now (b != 0), These are
+ // reads we've corrected with realignment, but they're
+ // also enriched for FPs so at high depth we reduce their
+ // confidence and let the depth do the talking. If it's
+ // real and deep, then we don't need every read aligning.
+ // We also reduce base quality too to reflect the
+ // chance of our realignment being incorrect.
+
+ seqQ = MIN(seqQ, seqQ/2 + 5); // q2p5
+
+ // Finally reduce indel quality.
+ // This is a blend of indelQ and base QUAL.
+ q = MIN((int)bam_get_qual(p->b)[p->qpos]/4+10, q/4+1);
+ }
+ }
+#endif
+
+ // Note baseQ changes some output fields such as I16, but has no
+ // significant affect on "call".
baseQ = p->aux>>8&0xff;
}
else
}
mapQ = p->b->core.qual < 255? p->b->core.qual : DEF_MAPQ; // special case for mapQ==255
if ( !mapQ ) r->mq0++;
+#ifdef GLF_DEBUG
+ fprintf(bcftools_stderr, "GLF2 %s\t%d\t%d\t%d,%d\n",
+ bam_get_qname(p->b), b, q,
+ seqQ, mapQ);
+#endif
if (q > seqQ) q = seqQ;
mapQ = mapQ < bca->capQ? mapQ : bca->capQ;
if (q > mapQ) q = mapQ;
for (i=0; i<4; i++) r->ADF[i] += lroundf((float)dp_ambig * r->ADF[i]/dp);
}
+ // Else consider downgrading bca->bases[] scores by AD vs AD_ref_missed
+ // ratios. This is detrimental on Illumina, but beneficial on PacBio CCS.
+ // It's possibly related to the homopolyer error likelihoods or overall
+ // Indel accuracy. Maybe tie this in to the -h option?
+
r->ori_depth = ori_depth;
// glfgen
errmod_cal(bca->e, n, 5, bca->bases, r->p); // calculate PL of each genotype
+
+ // TODO: account for the number of unassigned reads. If depth is 50,
+ // but AD is 5,7 then it may look like a variant but it probably
+ // should be low quality.
+
return n;
}
if ( bc->ori_ref < 0 )
{
bcf_update_info_flag(hdr, rec, "INDEL", NULL, 1);
- if ( fmt_flag&B2B_INFO_IDV )
- bcf_update_info_int32(hdr, rec, "IDV", &bca->max_support, 1);
- if ( fmt_flag&B2B_INFO_IMF )
- bcf_update_info_float(hdr, rec, "IMF", &bca->max_frac, 1);
+ uint32_t idv = bca->max_support;
+ if ( fmt_flag&B2B_INFO_IMF) {
+ float max_frac;
+ // Recompute IDV and IMF based on alignment results for more
+ // accurate counts, but only when in new "--indels-cns" mode.
+ if (bc->ADF && bc->ADR && bca->edlib) {
+ int max_ad = 0;
+ for (int k = 1; k < rec->n_allele; k++) {
+ if (max_ad < bc->ADF[k] + bc->ADR[k])
+ max_ad = bc->ADF[k] + bc->ADR[k];
+ }
+ max_frac = (double)(max_ad) / bc->ori_depth;
+ idv = max_ad;
+ } else {
+ max_frac = bca->max_frac;
+ }
+ // Copied here to maintain order for consistency of "make check"
+ if ( fmt_flag&B2B_INFO_IDV )
+ bcf_update_info_int32(hdr, rec, "IDV", &idv, 1);
+ bcf_update_info_float(hdr, rec, "IMF", &max_frac, 1);
+ } else {
+ if ( fmt_flag&B2B_INFO_IDV )
+ bcf_update_info_int32(hdr, rec, "IDV", &idv, 1);
+ }
}
bcf_update_info_int32(hdr, rec, "DP", &bc->ori_depth, 1);
if ( fmt_flag&B2B_INFO_ADF )
// for internal uses
int max_bases;
int indel_types[4]; // indel lengths
- int indel_win_size, indels_v20;
- int maxins, indelreg;
+ int indel_win_size, indels_v20, edlib;
+ int seqQ_offset; // edlib mode, seqQ=MIN(seqQ, offset - MIN(20,depth)*5);
+ int maxins, indelreg, poly_mqual;
int read_len;
char *inscns;
uint16_t *bases; // 5bit: unused, 6:quality, 1:is_rev, 4:2-bit base or indel allele (index to bcf_callaux_t.indel_types)
errmod_t *e;
void *rghash;
float indel_bias; // adjusts indel score threshold; lower => call more.
+ float del_bias; // (-.9 < x < .9) error profile; >0 => more del, <0 => more ins
+ float vs_ref; // 0 to 1. 0: score vs next-best. 1: score vs ref
int32_t *ref_nm, *alt_nm; // pointers to bcf_call_t.{ref_nm,alt_nm}
unsigned int nnm[2]; // number of nm observations
float nm[2]; // cumulative count of mismatches in ref and alt reads
const bcf_callaux_t *bca, const char *ref);
int bcf_call_gap_prep(int n, int *n_plp, bam_pileup1_t **plp, int pos, bcf_callaux_t *bca, const char *ref);
int bcf_iaux_gap_prep(int n, int *n_plp, bam_pileup1_t **plp, int pos, bcf_callaux_t *bca, const char *ref);
+ int bcf_edlib_gap_prep(int n, int *n_plp, bam_pileup1_t **plp, int pos,
+ bcf_callaux_t *bca, const char *ref, int ref_len);
+
void bcf_callaux_clean(bcf_callaux_t *bca, bcf_call_t *call);
int bcf_cgp_l_run(const char *ref, int pos);
int est_indelreg(int pos, const char *ref, int l, char *ins4);
+/* ----------------------------------------------------------------------
+ * Shared between bam2bcf_indel.c and bam2bcf_edlib.c
+ */
+
+// Take a reference position tpos and convert to a query position (returned).
+// This uses the CIGAR string plus alignment c->pos to do the mapping.
+//
+// *_tpos is returned as tpos if query overlaps tpos, but for deletions
+// it'll be either the start (is_left) or end (!is_left) ref position.
+int tpos2qpos(const bam1_core_t *c, const uint32_t *cigar, int32_t tpos, int is_left, int32_t *_tpos);
+
+// Identify spft-clip length, position in seq, and clipped seq len
+void get_pos(const bcf_callaux_t *bca, bam_pileup1_t *p,
+ int *sc_len_r, int *slen_r, int *epos_r, int *end);
+
+// Compute the consensus for this sample 's', minus indels which
+// get added later.
+char *bcf_cgp_calc_cons(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, int *types, int n_types,
+ int max_ins, int s);
+
#ifdef __cplusplus
}
#endif
--- /dev/null
+/* bam2bcf_indel.c -- indel caller.
+
+ Copyright (C) 2010, 2011 Broad Institute.
+ Copyright (C) 2012-2014,2016-2017, 2021-2024 Genome Research Ltd.
+
+ Author: Heng Li <lh3@sanger.ac.uk>
+ Petr Danecek <pd3@sanger.ac.uk>
+ James Bonfield <jkb@sanger.ac.uk>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE. */
+
+// Show consensus
+//#define CONS_DEBUG
+
+// Show alignments to consensus
+//#define ALIGN_DEBUG
+
+#include <assert.h>
+#include <ctype.h>
+#include <string.h>
+#include <math.h>
+#include <htslib/hts.h>
+#include <htslib/sam.h>
+#include <htslib/khash_str2int.h>
+#include "bam2bcf.h"
+#include "str_finder.h"
+
+#include <htslib/ksort.h>
+// Is there no way to share these between the 3 implementations?
+KSORT_INIT_STATIC_GENERIC(uint32_t)
+
+#define MINUS_CONST 0x10000000
+
+#define MAX_TYPES 64
+
+#ifndef MIN
+# define MIN(a,b) ((a)<(b)?(a):(b))
+#endif
+
+#ifndef ABS
+# define ABS(a) ((a)<0?-(a):(a))
+#endif
+
+#ifndef MAX
+# define MAX(a,b) ((a)>(b)?(a):(b))
+#endif
+
+// l is the relative gap length and l_run is the length of the homopolymer
+// on the reference.
+//
+// Larger seqQ is good, so increasing tandemQ calls more indels,
+// and longer l_run means fewer calls. It is capped later at 255.
+// For short l_runs, the qual is simply based on size of indel
+// larger ones being considered more likely to be real.
+// Longer indels get assigned a score based on the relative indel size
+// to homopolymer, where l_run base will have already been verified by
+// the caller to ensure it's compatible.
+static inline int est_seqQ(const bcf_callaux_t *bca, int l, int l_run, int str_len)
+{
+ int q, qh;
+ // Short indels are more likely sequencing error than large ones.
+ // So "seqQ" scales with size of observation "l".
+ //
+ // Note openQ and extQ are error likelihoods in Phred scale. Hence high
+ // openQ means we're very unlikely to miscall an indel.
+ // Ie it's not the open/ext "costs" normally used in alignment; more the reverse.
+ //
+ // We use MIN(q,qh) below, so we can remove the q component by specifying
+ // a large -o parameter in mpileup.
+ q = bca->openQ + bca->extQ * (abs(l) - 1);
+
+ // Orig method; best with Illumina (high openQ)
+// qh = bca->tandemQ * (double)abs(l) / l_run + .499;
+
+ // Penalise longer homopolymers quadratically more, but boost shorter ones.
+ // Best with CCS (low openQ)
+ //qh = 2 * bca->tandemQ * pow((double)abs(l) / l_run, 1.5) + .499;
+
+ // (l/l_run)^1.26 for openQ=25 or ^1 for openQ=40.
+// double openQ = MIN(40, bca->openQ);
+// qh = (30/openQ) * bca->tandemQ
+// * pow((double)abs(l) / l_run, 1/sqrt(openQ/40)) + .499;
+
+ // Linear scaled on openQ too
+ qh = bca->tandemQ * (double)abs(l) / l_run + .499;
+
+ // Generic maybe ?
+ // power = 1/sqrt(MIN(40,bca->openQ)/40.);
+ // qh = ... * pow((double)abs(l)/l_run, power)
+
+ // bam2bcf.c caps has "if q>seqQ) q=seqQ" so it caps base qual 'q'.
+ // A 1bp indel would therefore have a maximum qual it could be considered based
+ // on open+ext. Hence why openQ is phred score indicating if the base is real
+ // or an over/under-call. (high openQ means high trust in base)
+ return q < qh? q : qh;
+}
+
+// Part of bcf_call_gap_prep.
+//
+// Scans the pileup to identify all the different sizes of indels
+// present.
+// types[] returned is sorted by size, from smallest (maybe negative) to largest.
+//
+// Returns types and fills out n_types_r, max_rd_len_r and ref_type_r,
+// or NULL on error.
+static int *bcf_cgp_find_types(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, bcf_callaux_t *bca, const char *ref,
+ int *max_rd_len_r, int *n_types_r,
+ int *ref_type_r, int *N_r) {
+ int i, j, t, s, N, m, max_rd_len, n_types;
+ int n_alt = 0, n_tot = 0, indel_support_ok = 0;
+ uint32_t *aux;
+ int *types;
+
+ // N is the total number of reads
+ for (s = N = 0; s < n; ++s)
+ N += n_plp[s];
+
+ bca->max_support = bca->max_frac = 0;
+ aux = (uint32_t*) calloc(N + 1, 4);
+ if (!aux)
+ return NULL;
+
+ m = max_rd_len = 0;
+ aux[m++] = MINUS_CONST; // zero indel is always a type (REF)
+
+ // Fill out aux[] array with all the non-zero indel sizes.
+ // Also tally number with indels (n_alt) and total (n_tot).
+ for (s = 0; s < n; ++s) {
+ int na = 0, nt = 0;
+ for (i = 0; i < n_plp[s]; ++i) {
+ const bam_pileup1_t *p = plp[s] + i;
+ ++nt;
+ if (p->indel != 0) {
+ ++na;
+ aux[m++] = MINUS_CONST + p->indel;
+ }
+
+ // FIXME: cache me in pileup struct.
+ j = bam_cigar2qlen(p->b->core.n_cigar, bam_get_cigar(p->b));
+ if (j > max_rd_len) max_rd_len = j;
+ }
+ double frac = (double)na/nt;
+ if ( !indel_support_ok && na >= bca->min_support
+ && frac >= bca->min_frac )
+ indel_support_ok = 1;
+ if ( na > bca->max_support && frac > 0 )
+ bca->max_support = na, bca->max_frac = frac;
+
+ n_alt += na;
+ n_tot += nt;
+ }
+
+ // Sort aux[] and dedup
+ ks_introsort(uint32_t, m, aux);
+ for (i = 1, n_types = 1; i < m; ++i)
+ if (aux[i] != aux[i-1]) ++n_types;
+
+ // Taking totals makes it hard to call rare indels (IMF filter)
+ if ( !bca->per_sample_flt )
+ indel_support_ok = ( (double)n_alt / n_tot < bca->min_frac
+ || n_alt < bca->min_support )
+ ? 0 : 1;
+ if ( n_types == 1 || !indel_support_ok ) { // then skip
+ free(aux);
+ return NULL;
+ }
+
+ // Bail out if we have far too many types of indel
+ if (n_types >= MAX_TYPES) {
+ free(aux);
+ // TODO revisit how/whether to control printing this warning
+ if (hts_verbose >= 2)
+ fprintf(stderr, "[%s] excessive INDEL alleles at position %d. "
+ "Skip the position.\n", __func__, pos + 1);
+ return NULL;
+ }
+
+ // To prevent long stretches of N's to be mistaken for indels
+ // (sometimes thousands of bases), check the number of N's in the
+ // sequence and skip places where half or more reference bases are Ns.
+ int nN=0, i_end = pos + (2*bca->indel_win_size < max_rd_len
+ ?2*bca->indel_win_size : max_rd_len);
+ for (i=pos; i<i_end && ref[i]; i++)
+ nN += ref[i] == 'N';
+ if ( nN*2>(i-pos) ) {
+ free(aux);
+ return NULL;
+ }
+
+ // Finally fill out the types[] array detailing the size of insertion
+ // or deletion.
+ types = (int*)calloc(n_types, sizeof(int));
+ if (!types) {
+ free(aux);
+ return NULL;
+ }
+ t = 0;
+ for (i = 0; i < m; ++i) {
+ int sz = (int32_t)(aux[i] - MINUS_CONST);
+ int j;
+ for (j = i+1; j < m; j++)
+ if (aux[j] != aux[i])
+ break;
+
+ if (sz == 0
+ || (j-i >= bca->min_support &&
+ // Note, doesn't handle bca->per_sample_flt yet
+ (bca->per_sample_flt
+ || (double)(j-i) / n_tot >= bca->min_frac)))
+ types[t++] = sz;
+ i = j-1;
+ }
+ free(aux);
+
+ if (t <= 1) {
+ free(types);
+ return NULL;
+ }
+ n_types = t;
+
+ // Find reference type; types[?] == 0)
+ for (t = 0; t < n_types; ++t)
+ if (types[t] == 0) break;
+
+ *ref_type_r = t;
+ *n_types_r = n_types;
+ *max_rd_len_r = max_rd_len;
+ *N_r = N;
+
+ return types;
+}
+
+// Increment ins["str"] and freq["str"]
+#define NI 100 // number of alternative insertion sequences
+// Could use a hash table too, but expectation is a tiny number of alternatives
+typedef struct {
+ char *str[NI];
+ int len[NI];
+ int freq[NI];
+} str_freq;
+
+static int bcf_cgp_append_cons(str_freq *sf, char *str, int len, int freq) {
+ int j;
+
+ for (j = 0; j < NI && sf->str[j]; j++) {
+ if (sf->len[j] == len && memcmp(sf->str[j], str, len) == 0)
+ break;
+ }
+ if (j >= NI)
+ return 0; // too many choices; discard
+
+ sf->freq[j]+=freq;
+ if (!sf->str[j]) {
+ // new insertion
+ if (!(sf->str[j] = malloc(len+1)))
+ return -1;
+ memcpy(sf->str[j], str, len);
+ sf->len[j] = len;
+ }
+
+ return 0;
+}
+
+/*
+ * Compute the consensus for a specific indel type at pos.
+ *
+ * left_shift is the number of inserted(+) or deleted(-) bases added to
+ * the consensus before we get to pos. This is necessary so the alignment
+ * band is correct as it's expected to start at left/right edges in
+ * sync
+ *
+ * We accumulate into several buffers for counting base types:
+ * cons_base - consensus of data with p->indel == type, bases or gap
+ * ref_base - consensus of data with p->indel != type, bases or gap
+ * cons_ins - consensus of data with p->indel == type, insertions
+ * ref_ins - consensus of data with p->indel == type, bases or gap
+ *
+ * The purpose of cons_ins vs cons_base is if we have very low
+ * coverage due to nearly all reads being another type, then we can
+ * still get a robust consensus using the other data. If we don't
+ * have shallow data, then we'll not use as much of ref_base as we may
+ * have correlated variants.
+ *
+ * Eg:
+ * REF: AGCTATGAGGCTGATA
+ * SEQ: AGGTAGGAGGGTGATA (x1)
+ * SEQ: AGCTACGAGG*TGATA (x24)
+ * SEQ: AGCTACTAGG*TGATA (x24)
+ *
+ * Cons for no-del is Cs not Gs. Cannot trust it, so use N if shallow.
+ * CON: AGCTACNAGGGTGATA
+ *
+ * There are still some problems in cons_ins vs ref_ins assignment.
+ * We sometimes seem multiple similar-length insertions added at
+ * different locations. Ideally we'd like to consider these as all
+ * the same insertion if the size is the same and it's comparable seq.
+ */
+#define MAX_INS 8192
+static char **bcf_cgp_consensus(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, bcf_callaux_t *bca, const char *ref,
+ int ref_len, int left, int right,
+ int sample, int type, int biggest_del,
+ int *left_shift, int *right_shift,
+ int *band, int *tcon_len, int *cpos_pos,
+ int pos_l, int pos_r) {
+ // Map ASCII ACGTN* to 012345
+ static uint8_t base6[256] = {
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,5,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ //A C G *^ T
+ 4,0,4,1,4,4,4,2, 4,4,4,4,4,4,4,4, 4,4,4,4,3,3,4,4, 4,4,4,4,4,4,4,4,
+ 4,0,4,1,4,4,4,2, 4,4,4,4,4,4,4,4, 4,4,4,4,3,3,4,4, 4,4,4,4,4,4,4,4,
+
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ };
+
+ // single base or del
+ int (*cons_base)[6] = calloc(right - left + 1, sizeof(*cons_base));
+ // multi-base insertions
+ str_freq *cons_ins = calloc(right - left + 1, sizeof(*cons_ins));
+
+ // non-indel ref for all reads on this sample, rather than those just
+ // matching type. We use this for handling the case where we have a
+ // homozygous deletion being studied, but with 1 or 2 reads misaligned
+ // and containing a base there.
+ //
+ // Eg if the type[]=0 consensus is made up of a very small sample size,
+ // which is also enriched for highly error prone data. We can use
+ // the other reads from type[] != 0 to flesh out the consensus and
+ // improve accuracy.
+ int (*ref_base)[6] = calloc(right - left + 1, sizeof(*ref_base));
+ str_freq *ref_ins = calloc(right - left + 1, sizeof(*ref_ins));
+ int i, j, k, s = sample;
+ char **cons = NULL;
+
+ if (!cons_base || !cons_ins || !ref_base || !ref_ins)
+ goto err;
+
+ //--------------------------------------------------
+ // Accumulate sequences into cons_base and cons_ins arrays
+ int local_band_max = 0; // maximum absolute deviation from diagonal
+ int total_span_str = 0;
+ int type_depth = 0;
+ for (i = 0; i < n_plp[s]; i++) {
+ const bam_pileup1_t *p = plp[s] + i;
+ bam1_t *b = p->b;
+ int x = b->core.pos; // ref coordinate
+ int y = 0; // seq coordinate
+ uint32_t *cigar = bam_get_cigar(b);
+ uint8_t *seq = bam_get_seq(b);
+
+ int local_band = 0; // current deviation from diagonal
+ for (k = 0; k < b->core.n_cigar; ++k) {
+ int op = cigar[k] & BAM_CIGAR_MASK;
+ int len = cigar[k] >> BAM_CIGAR_SHIFT;
+ int base;
+ int skip_to = 0;
+
+ switch(op) {
+ case BAM_CSOFT_CLIP:
+ y += len;
+ break;
+
+ case BAM_CMATCH:
+ case BAM_CEQUAL:
+ case BAM_CDIFF: {
+ // Can short-cut this with j_start and j_end based on
+ // x+len and left,right
+ for (j = 0; j < len; j++, x++, y++) {
+ if (x < left) continue;
+ if (x >= right) break;
+
+ base = bam_seqi(seq, y);
+ if (p->indel == type)
+ // Convert 4-bit base ambig code to 0,1,2,3,4 range
+ cons_base[x-left][seq_nt16_int[base]]++;
+ else if (x != pos+1) // indel being assessed question
+ ref_base[x-left][seq_nt16_int[base]]++;
+ }
+ break;
+ }
+
+ case BAM_CINS: {
+ if (x >= left && x < right) {
+ local_band += p->indel;
+ if (local_band_max < local_band)
+ local_band_max = local_band;
+ }
+
+ char ins[MAX_INS];
+ for (j = 0; j < len; j++, y++) {
+ if (x < left) continue;
+ if (x >= right)
+ break;
+ base = bam_seqi(seq, y);
+ if (j < MAX_INS)
+ ins[j] = seq_nt16_int[base];
+ }
+
+ // Insertions come before a ref match.
+ // 5I 5M is IIIIIM M M M M events, not
+ // {IIIII,M} M M M M choice. So we need to include the
+ // next match in our sequence when choosing the consensus.
+ if (x >= left && x < right) {
+ int ilen = j<MAX_INS?j:MAX_INS;
+ if (p->indel == type /*&& x == pos+1*/) {
+ // Assume any ins of the same size is the same ins.
+ // (This rescues misaligned insertions.)
+ if (bcf_cgp_append_cons(&cons_ins[x-left], ins,
+ ilen, 1) < 0)
+ goto err;
+ type_depth += (x == pos+1);
+ } else if (x != pos+1){
+ if (bcf_cgp_append_cons(&ref_ins[x-left], ins,
+ ilen, 1) < 0)
+ goto err;
+ }
+ }
+ break;
+ }
+
+ case BAM_CDEL:
+ if (x >= left && x < right) {
+ local_band += p->indel;
+ if (local_band_max < -local_band)
+ local_band_max = -local_band;
+ }
+
+ // Maybe not perfect for I/D combos, but likely sufficient.
+ for (j = 0; j < len; j++, x++) {
+ if (x < left) continue;
+ if (x >= right) break;
+ if ((p->indel == type && !p->is_del) || // starts here
+ (p->indel == 0 && p->is_del && len == -type)) { // left
+ cons_base[x-left][5]++;
+ type_depth += (x == pos+1);
+ } else if (x+len <= pos+1 || (skip_to && x > skip_to))
+ ref_base[x-left][5]++;
+ else if (x <= pos && x+len > pos+1) {
+ // we have a deletion which overlaps pos, but
+ // isn't the same "type". We don't wish to
+ // include these as they may bias the
+ // evaluation by confirming against a
+ // secondary consensus produced with the other
+ // deletion. We set a marker for how long to
+ // skip adding to ref_base.
+ if (x > skip_to)
+ skip_to = x+len;
+ }
+ }
+ break;
+ }
+ }
+
+ if (b->core.pos <= pos_l && x >= pos_r)
+ total_span_str++;
+
+ // Also track the biggest deviation +/- from diagonal. We use
+ // this band observation in our BAQ alignment step.
+ if (*band < local_band_max)
+ *band = local_band_max;
+ }
+
+ //--------------------------------------------------
+ // Expand cons_base to include depth from ref_base/ref_ins
+ // Caveat: except at pos itself, where true ref is used if type != 0
+
+#if 1 // TEST 1
+ // We could retest this heuristic further maybe.
+ for (i = 0; i < right-left; i++) {
+ // Total observed depth
+ int t = cons_base[i][0] + cons_base[i][1] + cons_base[i][2] +
+ cons_base[i][3] + cons_base[i][4] + cons_base[i][5];
+ for (j = 0; j < NI; j++) {
+ if (!cons_ins[i].str[j])
+ break;
+ t += cons_ins[i].freq[j];
+ }
+
+ // Similarly for depth on the non-ALT calls (NB: not necessarily
+ // REF as maybe it's other ALTs).
+ int r = ref_base[i][0] + ref_base[i][1] + ref_base[i][2] +
+ ref_base[i][3] + ref_base[i][4] + ref_base[i][5];
+ for (j = 0; j < NI; j++) {
+ if (!ref_ins[i].str[j])
+ break;
+ r += ref_ins[i].freq[j];
+ }
+
+ // When evaluating this particular indel, we don't want to
+ // penalise alignments by SNP errors elsewhere. This can
+ // happen when we have low depth for a particular 'type'.
+ //
+ // So add in a little data from ref_base/ref_ins.
+ double rfract = (r - t*2)*.75 / (r+1);
+
+ if (rfract < 1.01 / (r+1e-10))
+ rfract = 1.01 / (r+1e-10); // low depth compensation
+// if (rfract > 0.2)
+// rfract = 0.2;
+
+ // TODO: consider limiting rfract so we never drown out the
+ // signal. We want to use the remaining data only to correct
+ // for sequencing errors in low depth alleles. If we get
+ // conflicts, it's better to use N than to change a base
+ // incase that variant is genuine.
+ if (i+left >= pos+1 && i+left < pos+1-biggest_del) {
+ // We're overlapping the current indel region, so
+ // we don't wish to bring in evidence from the other
+ // "type" data as it'll harm calling.
+ continue;
+ } else {
+ // Otherwise add in a portion of other data to
+ // boost low population numbers.
+ cons_base[i][0] += rfract * ref_base[i][0];
+ cons_base[i][1] += rfract * ref_base[i][1];
+ cons_base[i][2] += rfract * ref_base[i][2];
+ cons_base[i][3] += rfract * ref_base[i][3];
+ cons_base[i][4] += rfract * ref_base[i][4];
+ cons_base[i][5] += rfract * ref_base[i][5];
+ }
+
+ // Similarly for insertions too; consider a different rfract here?
+ for (j = 0; j < NI; j++) {
+ if (!ref_ins[i].str[j])
+ break;
+ if (bcf_cgp_append_cons(&cons_ins[i],
+ ref_ins[i].str[j], ref_ins[i].len[j],
+ rfract * ref_ins[i].freq[j]) < 0)
+ goto err;
+ }
+ }
+#endif
+
+ //--------------------------------------------------
+ // Allocate consensus buffer, to worst case length
+ int max_len = right-left;
+ for (i = 0; i < right-left; i++) {
+ if (!cons_ins[i].str[0])
+ continue;
+
+ int ins = 0;
+ for (j = 0; j < NI; j++) {
+ if (!cons_ins[i].str[j])
+ break;
+ if (cons_ins[i].str[j] && ins < cons_ins[i].len[j])
+ ins = cons_ins[i].len[j];
+ }
+ max_len += ins;
+ }
+ max_len += MAX(0, type); // incase type inserted bases never occur
+ cons = malloc((max_len+1)*2 + sizeof(char *)*2);
+ if (!cons)
+ goto err;
+ cons[0] = (char *)&cons[2];
+ cons[1] = cons[0] + max_len+1;
+
+ //--------------------------------------------------
+ // Merge insertions where they are the same length but different
+ // sequences.
+ // NB: we could just index by length and have accumulators for each,
+ // instead of storing separately and merging later (here).
+ // Ie str_freq.str is [NI][5] instead.
+ for (i = 0; i < right-left; i++) {
+ int ins[MAX_INS][5];
+ for (j = 0; j < NI; j++) {
+ if (!cons_ins[i].str[j])
+ break;
+
+ if (cons_ins[i].freq[j] == 0)
+ continue; // already merged
+
+ int l;
+ for (l = 0; l < cons_ins[i].len[j]; l++) {
+ // Append to relevant frequency counter, zero all others
+ ins[l][0] = ins[l][1] = ins[l][2] = ins[l][3] = ins[l][4] = 0;
+ uint8_t b = cons_ins[i].str[j][l];
+ ins[l][b] = cons_ins[i].freq[j];
+ }
+
+ // Merge other insertions of the same length to ins[] counters
+ for (k = j+1; k < NI; k++) {
+ if (!cons_ins[i].str[k])
+ break;
+ if (cons_ins[i].len[k] != cons_ins[i].len[j])
+ continue;
+ if (cons_ins[i].freq[k] == 0)
+ continue; // redundant?
+
+ // Merge str[j] and str[k]
+ for (l = 0; l < cons_ins[i].len[k]; l++) {
+ uint8_t b = cons_ins[i].str[k][l];
+ ins[l][b] += cons_ins[i].freq[k];
+ }
+ cons_ins[i].freq[j] += cons_ins[i].freq[k];
+ cons_ins[i].freq[k] = 0;
+ }
+
+ // Now replace ins[j] with the consensus insertion of this len.
+ for (l = 0; l < cons_ins[i].len[j]; l++) {
+ int max_v = 0, base = 0;
+ int tot = ins[l][0] + ins[l][1] + ins[l][2]
+ + ins[l][3] + ins[l][4];
+ if (max_v < ins[l][0]) max_v = ins[l][0], base = 0;
+ if (max_v < ins[l][1]) max_v = ins[l][1], base = 1;
+ if (max_v < ins[l][2]) max_v = ins[l][2], base = 2;
+ if (max_v < ins[l][3]) max_v = ins[l][3], base = 3;
+ if (max_v < ins[l][4]) max_v = ins[l][4], base = 4;
+
+ cons_ins[i].str[j][l] = (max_v > 0.6*tot) ? base : 4;
+ }
+ }
+ }
+
+#define CONS_CUTOFF .40 // % needed for base vs N
+#define CONS_CUTOFF_DEL .35 // % to include any het del
+#define CONS_CUTOFF2 .80 // % needed for gap in cons[1]
+#define CONS_CUTOFF_INC .35 // % to include any insertion cons[0]
+#define CONS_CUTOFF_INC2 .80 // % to include any insertion cons[1] HOM
+#define CONS_CUTOFF_INS .60 // and then 60% needed for it to be bases vs N
+
+ //--------------------------------------------------
+ // Walk through the frequency arrays to call the consensus.
+ // We produce cons[0] and cons[1]. Both include strongly
+ // homozygous indels. Both also include the indel at 'pos'.
+ // However for heterozygous indels we call the most likely event
+ // for cons[0] and the less-likely alternative in cons[1].
+ // TODO: a proper phase analysis so multiple events end up
+ // combining together into the correct consensus.
+ *left_shift = 0;
+ *right_shift = 0;
+ int cnum;
+
+ // Het call filled out in cnum==0 (+ve or -ve).
+ // Used in cnum==1 to do the opposite of whichever way we did before.
+ int heti[MAX_INS] = {0}, hetd[MAX_INS] = {0};
+
+ *cpos_pos = -1;
+ for (cnum = 0; cnum < 2; cnum++) {
+ for (i = k = 0; i < right-left; i++) {
+ // Location in consensus matching the indel itself
+ if (i >= pos-left+1 && *cpos_pos == -1)
+ *cpos_pos = k;
+
+ int max_v = 0, max_v2 = 0, max_j = 4, max_j2 = 4, tot = 0;
+ for (j = 0; j < 6; j++) {
+ // Top 2 consensus calls
+ if (max_v < cons_base[i][j]) {
+ max_v2 = max_v, max_j2 = max_j;
+ max_v = cons_base[i][j], max_j = j;
+ } else if (max_v2 < cons_base[i][j]) {
+ max_v2 = cons_base[i][j], max_j2 = j;
+ }
+ tot += cons_base[i][j];
+ }
+
+ // +INS
+ int max_v_ins = 0, max_j_ins = 0;
+ int tot_ins = 0;
+ for (j = 0; j < NI; j++) {
+ if (i+left==pos+1)
+ if (type > 0 && i+left == pos+1
+ && cons_ins[i].len[j] < type && j == 0) {
+ cons_ins[i].str[j] = realloc(cons_ins[i].str[j], type);
+ if (!cons_ins[i].str[j])
+ goto err;
+ memset(cons_ins[i].str[j] + cons_ins[i].len[j],
+ 4, type - cons_ins[i].len[j]);
+ cons_ins[i].len[j] = type;
+ }
+ if (!cons_ins[i].str[j])
+ break;
+ if (cons_ins[i].freq[j] == 0)
+ continue; // previously merged
+
+ if (max_v_ins < cons_ins[i].freq[j])
+ //if (i != pos-left+1 || cons_ins[i].len[j] == type)
+ max_v_ins = cons_ins[i].freq[j], max_j_ins = j;
+ tot_ins += cons_ins[i].freq[j];
+ }
+
+ // NB: tot is based on next matching base, so it includes
+ // everything with or without the insertion.
+ int tot_sum = tot;
+ int always_ins =
+ (i == pos-left+1 && type>0) || // current eval
+ max_v_ins > CONS_CUTOFF_INC2*tot_sum;// HOM
+ int het_ins = 0;
+ if (!always_ins && max_v_ins >= bca->min_support) {
+ // Candidate HET ins.
+ if (cnum == 0) {
+ het_ins = max_v_ins > CONS_CUTOFF_INC * tot_sum;
+ if (i < MAX_INS) heti[i] = het_ins
+ ? 1
+ : (max_v_ins > .3*tot_sum ? -1:0);
+ } else {
+ // HET but uncalled before
+ het_ins = i < MAX_INS ? (heti[i] == -1) : 0;
+ }
+ }
+
+ if (always_ins || het_ins) {
+ if (max_v_ins > CONS_CUTOFF_INS*tot_ins) {
+ // Insert bases
+ for (j = 0; j < cons_ins[i].len[max_j_ins]; j++) {
+ if (cnum == 0) {
+ if (k < pos-left+*left_shift)
+ (*left_shift)++;
+ else
+ (*right_shift)++;
+ }
+ cons[cnum][k++] = cons_ins[i].str[max_j_ins][j];
+ }
+ } else {
+ for (j = 0; j < cons_ins[i].len[max_j_ins]; j++)
+ cons[cnum][k++] = 4; // 'N';
+ }
+ }
+
+ // Call deletions & bases
+ int always_del = (type < 0 && i > pos-left && i <= pos-left-type)
+ || cons_base[i][5] > CONS_CUTOFF2 * tot; // HOM del
+ int het_del = 0;
+ if (!always_del && cons_base[i][5] >= bca->min_support) {
+ // Candidate HET del.
+ if (cnum == 0) {
+ int tot2 = tot;
+ if (i > pos-left && i <= pos-left-biggest_del)
+ tot2 = total_span_str - type_depth;
+ het_del = cons_base[i][5] >= CONS_CUTOFF_DEL * tot2;
+
+ if (i < MAX_INS) {
+ if (i > pos-left && i <= pos-left-biggest_del)
+ hetd[i] = 0;
+ else
+ hetd[i] = het_del
+ ? 1
+ : (cons_base[i][5] >= .3 * tot2 ? -1 : 0);
+ }
+ } else {
+ // HET del uncalled on cnum 0
+ het_del = i < MAX_INS ? (hetd[i] == -1) : 0;
+ if (max_j == 5 && het_del == 0) {
+ max_v = max_v2;
+ max_j = max_j2;
+ }
+ }
+ }
+ if (always_del || het_del) {
+ // Deletion
+ if (k < pos-left+*left_shift)
+ (*left_shift)--;
+ else
+ (*right_shift)++;
+ } else {
+ // Finally the easy case - a non-indel base or an N
+ if (max_v > CONS_CUTOFF*tot)
+ cons[cnum][k++] = max_j; // "ACGTN*"
+ else if (max_v > 0)
+ cons[cnum][k++] = 4; // 'N';
+ else {
+ cons[cnum][k] = left+k < ref_len
+ ? base6[(uint8_t)ref[left+k]]
+ : 4;
+ k++;
+ }
+ }
+ }
+
+ tcon_len[cnum] = k;
+ }
+
+ // TODO: replace by io_lib's string pool for rapid tidying.
+ // For now this isn't the bottleneck though.
+ for (i = 0; i < right-left; i++) {
+ for (j = 0; j < NI; j++) {
+ if (cons_ins[i].str[j])
+ free(cons_ins[i].str[j]);
+ if (ref_ins[i].str[j])
+ free(ref_ins[i].str[j]);
+ }
+ }
+
+ err:
+ free(cons_base);
+ free(ref_base);
+ free(cons_ins);
+ free(ref_ins);
+
+ return cons;
+}
+
+// A rename of bcf_cgp_calc_cons from bam2bcf_indel.c
+//
+// Compute the insertion consensus for this sample 's' via a basic
+// majority rule.
+//
+// TODO: merge this into bcf_cgp_consensus as another return value?
+static char *bcf_cgp_calc_ins_cons(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, int *types, int n_types,
+ int max_ins, int s) {
+ return bcf_cgp_calc_cons(n, n_plp, plp, pos, types, n_types, max_ins, s);
+}
+
+#define MAX(a,b) ((a)>(b)?(a):(b))
+#define MIN(a,b) ((a)<(b)?(a):(b))
+
+// Compile with LIBS="-L. -ldl -ledlib" CLD=g++
+
+// This is faster than ksw and BAQ, meaning we can use larger --indel-size and
+// get a more accurate context, improving alignments further. This *may*
+// compensate for reduced sensitivity.
+#include "edlib.h"
+int edlib_glocal(uint8_t *ref, int l_ref, uint8_t *query, int l_query,
+ double m, double del_bias)
+{
+ EdlibAlignConfig cfg =
+ edlibNewAlignConfig(
+ //ABS(type)+ABS(l_ref-l_query)+10,
+ -1, // k; use small positive for faster alignment
+ EDLIB_MODE_HW, // mode
+#ifdef ALIGN_DEBUG
+ EDLIB_TASK_PATH,
+#else
+ EDLIB_TASK_LOC,
+#endif
+ NULL, // additionalEqualities
+ 0); // additionalEqualitiesLength
+ EdlibAlignResult r =
+ edlibAlign((char *)query, l_query, (char *)ref, l_ref, cfg);
+
+ if (r.status != EDLIB_STATUS_OK || r.numLocations < 1 ||
+ !r.endLocations || !r.startLocations) {
+ edlibFreeAlignResult(r);
+ return INT_MAX;
+ }
+
+#ifdef ALIGN_DEBUG
+ // NB: Needs linking against the C++ libedlib.a as our cut-down C
+ // implementation misses the alignment generation code.
+ {
+ int i, j = 0, pt = r.startLocations[0], pq = 0;
+ char line1[80];
+ char line2[80];
+ char line3[80];
+ for (i = 0; i < r.alignmentLength && pt < r.endLocations[0]; i++) {
+ int n;
+ switch (n = r.alignment[i]) {
+ case 0: // match
+ case 3: // mismatch
+ line1[j] = "ACGTN"[ref[pt++]];
+ line2[j] = "ACGTN"[query[pq++]];
+ line3[j] = " x"[n==3];
+ break;
+ case 2: // insertion to ref
+ line1[j] = "ACGTN"[ref[pt++]];
+ line2[j] = '-';
+ line3[j] = '-';
+ break;
+ case 1: // insertion to query
+ line1[j] = '-';
+ line2[j] = "ACGTN"[query[pq++]];
+ line3[j] = '+';
+ break;
+ }
+
+ if (++j == sizeof(line1)) {
+ fprintf(stderr, "%.*s\n", j, line1);
+ fprintf(stderr, "%.*s\n", j, line2);
+ fprintf(stderr, "%.*s\n", j, line3);
+ j = 0;
+ }
+ }
+ if (j) {
+ fprintf(stderr, "%.*s\n", j, line1);
+ fprintf(stderr, "%.*s\n", j, line2);
+ fprintf(stderr, "%.*s\n", j, line3);
+ }
+ }
+#endif
+
+ // Aligned target length minus query length is an indication of the number
+ // of insertions and/or deletions.
+ //
+ // For CIGAR 10M1I10M t_len > l_query ("AC" / "ATC")
+ // For CIGAR 10M1D10M t_len < l_query ("ATC" / "AC")
+ // Hence t_len-l_query is -ve for net insertions and +ve for net deletions.
+ // If we compute nins and ndel directly via walking though EDLIB_TASK_PATH
+ // we'll see t_len-l_query == ndel-nins.
+ //
+ // If a technology has a significantly higher chance of making deletion
+ // errors than insertion errors, then we would view deletions as less
+ // indicative of this sequence not coming from this candidate allele than
+ // if it had insertion (as the deletions are more likely to be errors
+ // rather than real, relative to the insertions). Hence we can skew the
+ // score by the net delta of num_del - num_ins.
+ //
+ // Note this is an approximation that doesn't account for multiple
+ // insertions and deletions within the same sequence, but it is much faster
+ // as it doesn't require EDLIB_TASK_PATH to be computed.
+ //
+ // Given editDistance is +1 for every mismatch, insertion and deletion,
+ // provided the t_len-l_query multiplier < 1 then this is always +ve.
+
+ int t_len = *r.endLocations - *r.startLocations + 1;
+ int score = m*(r.editDistance - del_bias*(t_len - l_query));
+
+ edlibFreeAlignResult(r);
+ return score;
+}
+
+// Part of bcf_call_gap_prep.
+//
+// Realign using BAQ to get an alignment score of a single read vs
+// a haplotype consensus. TODO: replace BAQ with something more robust.
+//
+// There are many coordinates, so let's explain them.
+// - left, right, tbeg, tend, r_start and r_end are in aligned reference
+// coordinates.
+// left/right start from pos +/- indel_win_size.
+// r_start/r_end are the BAM first and last mapped coord on the reference.
+// tbeg and tend are the intersection of the two.
+// - qbeg and qend are in BAM sequence coordinates
+// - qpos is in sequence coordinates, relative to qbeg.
+//
+// To see what this means, we have illustrations with coordinates
+// above the seqs in reference space and below the seqs in BAM seq space.
+//
+// Overlap left:
+// tbeg tend
+// r_start left pos r_end right
+// REF :..............|--------------------#------:--------------|...
+// SEQ :..............|--------------------#------|
+// 0 qbeg qpos qend
+//
+// Overlap right:
+// r_start tend
+// left tbeg pos right r_end
+// REF ...|--------------:-----#---------------------|...........:
+// SEQ |-----#---------------------|...........:
+// qbeg qpos qend
+// 0
+//
+// The "-" sequence is the bit passed in.
+// Ie ref2 spans left..right and query spans qbeg..qend.
+// We need to adjust ref2 therefore to tbeg..tend.
+//
+// Fills out score
+// Returns 0 on success,
+// <0 on error
+static int bcf_cgp_align_score(bam_pileup1_t *p, bcf_callaux_t *bca,
+ int type, int band,
+ uint8_t *ref1, uint8_t *ref2, uint8_t *query,
+ int r_start, int r_end,
+ int tbeg, int tend1, int tend2,
+ int left, int right,
+ int qbeg, int qend,
+ int pos, int qpos, int max_deletion,
+ double qavg, double del_bias, int *score,
+ int *str_len1_p, int *str_len2_p) {
+ int atype = abs(type);
+ int l, sc1, sc2;
+
+ // Trim poly_Ns at ends of ref.
+ // This helps to keep len(ref) and len(query) similar, to reduce
+ // band size and reduce the chance of -ve BAQ scores.
+ for (l = 0; l < tend1-tbeg && l < tend2-tbeg; l++)
+ if (ref1[l + tbeg-left] != 4 || ref2[l + tbeg-left] != 4)
+ break;
+ if (l > atype)
+ tbeg += l-atype;
+
+ for (l = tend1-tbeg-1; l >= 0; l--)
+ if (ref1[l + tbeg-left] != 4)
+ break;
+ l = tend1-tbeg-1 - l;
+ if (l > atype)
+ tend1 -= l-atype;
+
+ for (l = tend2-tbeg-1; l >= 0; l--)
+ if (ref2[l + tbeg-left] != 4)
+ break;
+ l = tend2-tbeg-1 - l;
+ if (l > atype) {
+ tend2 -= l-atype;
+ }
+
+ // The bottom 8 bits are length-normalised score while
+ // the top bits are unnormalised.
+ //
+ // Try original cons and new cons and pick best.
+ // This doesn't reduce FN much (infact maybe adds very slightly),
+ // but it does reduce GT errors and is a slight reduction to FP.
+
+ double mm = 30; // a const average qual for now. Could tune
+ sc2 = edlib_glocal(ref2 + tbeg - left, tend2 - tbeg,
+ query, qend - qbeg, mm, del_bias);
+
+ if (tend1 != tend2 ||
+ memcmp((char *)ref1 + tbeg - left, (char *)ref2 + tbeg - left,
+ tend1 - tbeg) != 0)
+ sc1 = edlib_glocal(ref1 + tbeg - left, tend1 - tbeg,
+ query, qend - qbeg, mm, del_bias);
+ else
+ sc1 = INT_MAX; // skip
+
+ // Find the best of the two alignments
+ if (sc1 < 0 && sc2 < 0) {
+ *score = 0xffffff;
+ return 0;
+ }
+ if (sc1 < 0) {
+ // sc2 is already correct
+ } else if (sc2 < 0) {
+ sc2 = sc1;
+ } else {
+ // sc1 and sc2 both pass, so use best
+ if (sc2 > sc1)
+ sc2 = sc1;
+ }
+
+ // Sc is overall alignment score, in top 24 bits (SeqQ). It's based
+ // purely on the scores for the whole alignment.
+ // We also have a separate indel score in bottom 8 bits (IndelQ).
+ // This is a function of all sorts of attributes of the candidate indel
+ // itself, such as STR length and the presence of poor quality bases.
+
+ // Used for adjusting indelQ below. Lower l is more likely to call
+ // (--FN, ++FP). (NB CLI --indel_bias is 1/indel_bias var).
+ // Starts as average score per base, and then adjusted based on seq
+ // complexity / quality.
+
+ l = .5*(100. * sc2 / (qend - qbeg) + .499);
+
+ *score = (sc2<<8) | (int)MIN(255, l * bca->indel_bias * .5);
+
+ return 0;
+}
+
+// Part of bcf_call_gap_prep.
+//
+// Returns n_alt on success
+// -1 on failure
+
+// TODO: almost identical to bam2bcf_indel.c's copy, so we could share
+// the code and add a check on bca->edlib.
+static int bcf_cgp_compute_indelQ(int n, int *n_plp, bam_pileup1_t **plp,
+ bcf_callaux_t *bca, char *inscns,
+ int l_run, int max_ins,
+ int ref_type, int *types, int n_types,
+ double qavg, int *score,
+ int str_len1, int str_len2) {
+ // FIXME: n_types has a maximum; no need to alloc - use a #define?
+ int sc[MAX_TYPES], sumq[MAX_TYPES], s, i, j, t, K, n_alt, tmp;
+ memset(sumq, 0, n_types * sizeof(int));
+ int sum_indelQ1[100] = {0}; // n
+ int sum_indelQ2[100] = {0}; // n
+
+ // Confusing variable naming and bit usage.
+ //
+ // score[] is low 8 bits normalised (by len) alignment score
+ // top 24 bits full alignment score
+ // This gets cast into "sct"; mnemonic score-per-indel-type.
+ //
+ // sc = (score<<6) | type (index to types[] array for indel size)
+ // So sc>>14 = score>>(14-6) = score>>8. Ie full alignment score
+ for (s = K = 0; s < n; ++s) {
+ for (i = 0; i < n_plp[s]; ++i, ++K) {
+ bam_pileup1_t *p = plp[s] + i;
+ // Labelling is confusing here.
+ // sct is short for score.
+ // sc is score + t(type)
+ // Why aren't these variable names reversed?
+ int *sct = &score[K*n_types], seqQ, indelQ1=0, indelQ2=0, indelQ=0;
+ for (t = 0; t < n_types; ++t) sc[t] = sct[t]<<6 | t;
+ for (t = 1; t < n_types; ++t) // insertion sort
+ for (j = t; j > 0 && sc[j] < sc[j-1]; --j)
+ tmp = sc[j], sc[j] = sc[j-1], sc[j-1] = tmp;
+
+#ifdef ALIGN_DEBUG
+ fprintf(stderr, "READ %s\tscores ", bam_get_qname(p->b));
+ for (t = 0; t < n_types; ++t) {
+ fprintf(stderr, "%+2d/%-3d ", types[sc[t]&0x3f], sc[t]>>14);
+ }
+#endif
+
+ /* errmod_cal() assumes that if the call is wrong, the
+ * likelihoods of other events are equal. This is about
+ * right for substitutions, but is not desired for
+ * indels. To reuse errmod_cal(), I have to make
+ * compromise for multi-allelic indels.
+ */
+ if ((sc[0]&0x3f) == ref_type) {
+ // sc >> 14 is the total score. It's been shifted by 8
+ // from normalised score and 6 from type.
+ // &0x3f is type number
+
+ // Best call is REF. Compare vs best indel
+ indelQ = (sc[1]>>14) - (sc[0]>>14);
+ seqQ = est_seqQ(bca, types[sc[1]&0x3f], l_run, str_len1);
+ } else {
+ // look for the reference type
+ for (t = 0; t < n_types; ++t) {
+ if ((sc[t]&0x3f) == ref_type)
+ break;
+ }
+ indelQ = indelQ1 = (sc[t]>>14) - (sc[0]>>14);
+// fprintf(stderr, "IndelQ = %d: %d-%d",
+// indelQ, (sc[t]>>14), (sc[0]>>14));
+
+ // Best call is non-ref, compare vs next best non-ref,
+ // or ref if it's just 2 choices (most common case).
+ for (t = 1; t < n_types; t++)
+ if ((sc[t]&0x3f) == ref_type)
+ continue;
+ else break;
+ if (t == n_types)
+ t--; // it's ref, but it'll do as next best.
+ indelQ2 = (sc[t]>>14) - (sc[0]>>14);
+ seqQ = est_seqQ(bca, types[sc[0]&0x3f], l_run, str_len1);
+
+#if 1 // TEST 3
+ indelQ = bca->vs_ref*indelQ1 + (1-bca->vs_ref)*indelQ2;
+#endif
+ }
+
+ // So we lower qual in some, but raise the average to keep FN/FP
+ // ratios up.
+ // Is this key diff for PacBio old vs new HiFi?
+ indelQ /= bca->indel_bias*0.5;
+ indelQ1 /= bca->indel_bias*0.5;
+
+ // Or maybe just *2 if bca->poly_mqual and be done with it?
+ // Or perhaps adjust the MIN(qavg/20, ...) to MIN(qavg/10) ?
+
+ // Skew SeqQ and IndelQ based on a portion of the minimum quality
+ // found within a homopolymer. This is useful where the quality
+ // values are a bit mutable and move around in such data, but less
+ // so on clocked sequencing technologies.
+ //
+ // Enabling this causes lots of GT errors on Illumina.
+ // However on PacBio it's key to removal of false positives.
+ // ONT and UG seem somewhere inbetween.
+ if (bca->poly_mqual) { // TEST 4
+ int qpos = p->qpos, l;
+ uint8_t *seq = bam_get_seq(p->b);
+ uint8_t *qual = bam_get_qual(p->b);
+ int min_q = qual[qpos];
+
+ // scan homopolymer left
+ char baseL = bam_seqi(seq, qpos+1 < p->b->core.l_qseq
+ ? qpos+1 : qpos);
+ for (l = qpos; l >= 0; l--) {
+ if (bam_seqi(seq, l) != baseL)
+ break;
+ if (min_q > qual[l])
+ min_q = qual[l];
+ }
+
+ // scan homo-polymer right (including site of indel)
+ char base = bam_seqi(seq, qpos+1);
+ for (l = qpos+1; l < p->b->core.l_qseq; l++) {
+ if (min_q > qual[l])
+ min_q = qual[l];
+ if (bam_seqi(seq, l) != base)
+ break;
+ }
+
+ // We reduce -h so homopolymers get reduced likelihood of being
+ // called, but then optionally increase or decrease from there
+ // based on base quality. Hence lack of low quality bases in
+ // homopolymer will rescue the score back again, reducing FNs.
+
+ // The score factors here may also be machine specific, but for
+ // now these work well (tuned on PB HiFi).
+ seqQ += MIN(qavg/20, min_q - qavg/10);
+ indelQ += MIN(qavg/20, min_q - qavg/5);
+ indelQ1+= MIN(qavg/20, min_q - qavg/5);
+
+ if (seqQ < 0) seqQ = 0;
+ if (indelQ < 0) indelQ = 0;
+ if (indelQ1< 0) indelQ1= 0;
+ }
+
+ // This is the length-normalised score from bcf_cgp_align_score
+ tmp = sc[0]>>6 & 0xff;
+
+ // reduce indelQ
+ // high score = bad, low score = good; flip for indelQ
+ // low normalised scores leave indelQ unmodified
+ // high normalised scores set indelQ to 0
+ // inbetween scores have a linear scale from indelQ to 0
+// Altering the MAGIC value below (originally 111, but chosen for unknown
+// reasons) is comparable to altering --indel-bias.
+#define TMP_MAGIC 255.0
+
+ indelQ = tmp > TMP_MAGIC? 0 : (int)((1. - tmp/TMP_MAGIC) * indelQ + .499);
+ indelQ1= tmp > TMP_MAGIC? 0 : (int)((1. - tmp/TMP_MAGIC) * indelQ1+ .499);
+
+ indelQ = MIN(indelQ, 255);
+ indelQ1 = MIN(indelQ1, 255);
+
+ // Doesn't really help accuracy, but permits -h to take
+ // affect still.
+ if (indelQ > seqQ) indelQ = seqQ;
+ if (indelQ > 255) indelQ = 255;
+ if (indelQ1> 255) indelQ1= 255;
+ if (seqQ > 255) seqQ = 255;
+
+ // Use 22 bits in total.
+ // 0-7 IndelQ
+ // 8-15 SeqQ
+ // 16-22 Score-per-base
+ p->aux = (sc[0]&0x3f)<<16 | seqQ<<8 | indelQ;
+ sumq[sc[0]&0x3f] += indelQ;
+
+#ifdef ALIGN_DEBUG
+ fprintf(stderr, "\t%d\t%d\n", indelQ, seqQ);
+#endif
+
+ // Experiment in p->aux vs sumq.
+ // One gives likelihood of an indel being here, while the other
+ // is likelihood of a specific genotype? But which is which?
+
+ sum_indelQ1[s] += indelQ1;
+ sum_indelQ2[s] += indelQ;
+ }
+ }
+
+ // Determine bca->indel_types[] and bca->inscns.
+ // Sumq[0] is always reference.
+ // Sumq[1] is best non-ref (and maybe better than ref)
+ bca->maxins = max_ins;
+ bca->inscns = (char*) realloc(bca->inscns, bca->maxins * 4);
+ if (bca->maxins && !bca->inscns)
+ return -1;
+ for (t = 0; t < n_types; ++t)
+ sumq[t] = sumq[t]<<6 | t;
+ for (t = 1; t < n_types; ++t) // insertion sort
+ for (j = t; j > 0 && sumq[j] > sumq[j-1]; --j)
+ tmp = sumq[j], sumq[j] = sumq[j-1], sumq[j-1] = tmp;
+ for (t = 0; t < n_types; ++t) // look for the reference type
+ if ((sumq[t]&0x3f) == ref_type) break;
+
+ if (t) { // then move the reference type to the first
+ tmp = sumq[t];
+ for (; t > 0; --t) sumq[t] = sumq[t-1];
+ sumq[0] = tmp;
+ }
+
+ for (t = 0; t < 4; ++t) bca->indel_types[t] = B2B_INDEL_NULL;
+ for (t = 0; t < 4 && t < n_types; ++t) {
+ bca->indel_types[t] = types[sumq[t]&0x3f];
+#ifdef ALIGN_DEBUG
+ fprintf(stderr, "TYPE %+2d %d\n", types[t], sumq[t]>>6);
+#endif
+ if (bca->maxins) // potentially an insertion
+ memcpy(&bca->inscns[t * bca->maxins],
+ &inscns[(sumq[t]&0x3f) * max_ins], bca->maxins);
+ }
+
+ // Update p->aux.
+ // If per-alignment type isn't found, then indelQ/seqQ is 0,
+ // otherwise unchanged.
+ for (s = n_alt = 0; s < n; ++s) {
+ for (i = 0; i < n_plp[s]; ++i) {
+ bam_pileup1_t *p = plp[s] + i;
+ int x = types[p->aux>>16&0x3f];
+ for (j = 0; j < 4; ++j)
+ if (x == bca->indel_types[j]) break;
+ p->aux = j<<16 | (j == 4? 0 : (p->aux&0xffff));
+ if ((p->aux>>16&0x3f) > 0) ++n_alt;
+#ifdef ALIGN_DEBUG
+ fprintf(stderr, "FIN %s\t%d\t%d\t%d\n",
+ bam_get_qname(p->b), (p->aux>>16)&0x3f,
+ bca->indel_types[(p->aux>>16)&0x3f], p->aux&0xff);
+#endif
+ }
+ }
+
+ return n_alt;
+}
+
+/*
+FIXME: with high number of samples, do we handle IMF correctly? Is it
+fraction of indels across entire data set, or just fraction for this
+specific sample? Needs to check bca->per_sample_flt (--per-sample-mF) opt.
+ */
+
+/*
+ notes:
+ - n .. number of samples
+ - the routine sets bam_pileup1_t.aux of each read as follows:
+ - 6: unused
+ - 6: the call; index to bcf_callaux_t.indel_types .. (aux>>16)&0x3f
+ - 8: estimated sequence quality .. (aux>>8)&0xff
+ - 8: indel quality .. aux&0xff
+ */
+int bcf_edlib_gap_prep(int n, int *n_plp, bam_pileup1_t **plp, int pos,
+ bcf_callaux_t *bca, const char *ref, int ref_len)
+{
+ if (ref == 0 || bca == 0) return -1;
+
+ int i, s, t, n_types, *types = NULL, max_rd_len, left, right, max_ins;
+ int *score = NULL;
+ int N, K, l_run, ref_type, n_alt = -1;
+ char *inscns = NULL, *query = NULL;
+
+ // determine if there is a gap
+ for (s = N = 0; s < n; ++s) {
+ for (i = 0; i < n_plp[s]; ++i)
+ if (plp[s][i].indel != 0) break;
+ if (i < n_plp[s]) break;
+ }
+ if (s == n)
+ // there is no indel at this position.
+ return -1;
+
+ // Find average base quality over this region
+ double qavg = 30, qsum = 0, qcount = 0;
+ int qmax = 0;
+ for (s = 0; s < n; s++) {
+ for (i = 0; i < n_plp[s]; i++) {
+#define QWIN 50
+ bam_pileup1_t *p = plp[s] + i;
+ int kstart = p->qpos - QWIN > 0 ? p->qpos - QWIN : 0;
+ int kend = p->qpos + QWIN < p->b->core.l_qseq
+ ? p->qpos + QWIN : p->b->core.l_qseq;
+ uint8_t *qual = bam_get_qual(p->b);
+ int k;
+ for (k = kstart; k < kend; k++) {
+ qsum += qual[k];
+ qcount++;
+ if (qmax < qual[k])
+ qmax = qual[k];
+ }
+ }
+ }
+ qavg = (qsum+1) / (qcount+1);
+
+ // find out how many types of indels are present
+ types = bcf_cgp_find_types(n, n_plp, plp, pos, bca, ref,
+ &max_rd_len, &n_types, &ref_type, &N);
+ if (!types)
+ goto err;
+
+
+ // calculate left and right boundary, based on type size for a bit more
+ // speed.
+ int max_indel = 20*MAX(ABS(types[0]), ABS(types[n_types-1]))
+ + bca->indel_win_size/4;
+ if (max_indel > bca->indel_win_size)
+ max_indel = bca->indel_win_size;
+ left = pos > max_indel ? pos - max_indel : 0;
+ right = pos + max_indel;
+
+ int del_size = types[0]<0 ? -types[0] : 0;
+ right += del_size;
+
+ // in case the alignments stand out the reference
+ for (i = pos; i < right; ++i)
+ if (ref[i] == 0) break;
+ right = i;
+
+ // compute the likelihood given each type of indel for each read
+ max_ins = types[n_types - 1]; // max_ins is at least 0
+
+ // The length of the homopolymer run around the current position
+ l_run = bcf_cgp_l_run(ref, pos);
+ int l_run_base = seq_nt16_table[(uint8_t)ref[pos+1]];
+ int l_run_ins = 0;
+
+ // construct the consensus sequence (minus indels, which are added later)
+ if (max_ins > 0) {
+ // TODO: replace filling inscns[] with calc_consensus return
+ // so the merges of the insertion consensus for type[t] is
+ // reported directly. (It may need adjustment to avoid N)
+ inscns = bcf_cgp_calc_ins_cons(n, n_plp, plp, pos,
+ types, n_types, max_ins, s);
+ if (!inscns)
+ return -1;
+ }
+
+ query = (char*) calloc(right - left + max_rd_len + max_ins + 2, 1);
+ score = (int*) calloc(N * n_types, sizeof(int));
+ bca->indelreg = 0;
+ double nqual_over_60 = bca->nqual / 60.0;
+
+ int biggest_del = 0;
+ int biggest_ins = 0;
+ for (t = 0; t < n_types; t++) {
+ if (biggest_del > types[t])
+ biggest_del = types[t];
+ if (biggest_ins < types[t])
+ biggest_ins = types[t];
+ }
+ int band = biggest_ins - biggest_del; // NB del is -ve
+
+ // Find left & right extents of STR covering pos, from ref
+ int pos_l = pos, pos_r = pos;
+ {
+ rep_ele *reps, *elt, *tmp;
+ int pstart = MAX(0, pos-30);
+ int pmid = pos-pstart;
+ int pend = MIN(ref_len, pos+30);
+ reps = find_STR((char *)&ref[pstart], pend-pstart, 0);
+ DL_FOREACH_SAFE(reps, elt, tmp) {
+ if (elt->end >= pmid && elt->start <= pmid) {
+ if (pos_l > pstart + elt->start)
+ pos_l = pstart + elt->start;
+ if (pos_r < pstart + elt->end)
+ pos_r = pstart + elt->end;
+ }
+ DL_DELETE(reps, elt);
+ free(elt);
+ }
+ }
+
+ int str_len1 = l_run, str_len2 = l_run/4;
+ for (t = 0; t < n_types; ++t) {
+ int l, ir;
+
+ // Compute indelreg. This is the context in the reference. Eg:
+ //
+ // REF: AG--TTTC Inscns is "TT".
+ // SEQ: AGTTTTTC Indelreg is 3; next 3 "TTT" bases
+ //
+ // => GTTT GTTTTT is call.
+ if (types[t] == 0)
+ ir = 0;
+ else if (types[t] > 0)
+ ir = est_indelreg(pos, ref, types[t], &inscns[t*max_ins]);
+ else
+ ir = est_indelreg(pos, ref, -types[t], 0);
+
+ if (ir > bca->indelreg)
+ bca->indelreg = ir;
+
+ // Realignment score, computed via BAQ
+ for (s = K = 0; s < n; ++s) {
+ char **tcons;
+ int left_shift, right_shift;
+ int tcon_len[2];
+ int cpos_pos;
+ tcons = bcf_cgp_consensus(n, n_plp, plp, pos, bca, ref, ref_len,
+ left, right, s, types[t], biggest_del,
+ &left_shift, &right_shift, &band,
+ tcon_len, &cpos_pos, pos_l, pos_r);
+ // TODO: Consensus for a deletion shouldn't match the
+ // consensus for type 0. Eg consider
+ // vv vv
+ // REF: AATGTGTGAACAA REF: AATGTG--AACAA
+ // T0: AATGTG--AACAA T0: AATGTG--AACAA
+ // T-2: AA--TGTGAATAA T-2: AA--TGTGAATAA:
+ //
+ // On left: both T0 and T-2 are the same length, as it's
+ // just a deletion that moved. We may end up assigning
+ // reads to an indel allele based on the SNP they have and
+ // not the actual indel.
+ // There *is* a deletion here though, but only 1. How do
+ // we call it once only? Need to replace entire region
+ // with a reassembly.
+ //
+ // On right: T0 and T-2 have same length again, but there
+ // isn't an indel as it's ins+del vs del+ins. They're
+ // also the same length as the REF for this region.
+ // Hence likelihood of this variant existing is tied in
+ // with their equal and high similarity with/to the ref.
+ //
+ // We could do an alignment of tcons[0] and tcons[1] and check
+ // whether their differences are consistent with (ie the
+ // hamming distance is at least ABS(types[t]/2). I don't think
+ // it'll rescue many FPs though.
+
+#ifdef CONS_DEBUG
+ {
+ int j;
+ for (j = 0; j < 2; j++) {
+ int k;
+ fprintf(stderr, "Cons%d @ %d %4d/%4d ",
+ j, pos, types[t], left_shift);
+ for (k = 0; k < tcon_len[j]; k++) {
+ if (k == cpos_pos)
+ putc('#', stderr);
+ putc("ACGTN"[(uint8_t)tcons[j][k]], stderr);
+ }
+ putc('\n', stderr);
+ }
+ }
+#endif
+
+ // Scan for base-runs in the insertion.
+ // We use this to avoid over-correction in est_seqQ when the
+ // insertion is not part of the neighbouring homopolymer.
+ int k = tcons[0][cpos_pos], j;
+ for (j = 0; j < types[t]; j++)
+ if (tcons[0][cpos_pos+j] != k)
+ break;
+ if (j && j == types[t])
+ l_run_ins |= "\x1\x2\x4\x8\xf"[k]; // ACGTN
+ if (types[t] < 0)
+ l_run_ins |= 0xff;
+
+ // align each read to consensus(es)
+ for (i = 0; i < n_plp[s]; ++i, ++K) {
+ bam_pileup1_t *p = plp[s] + i;
+
+ // Some basic ref vs alt stats.
+ int imq = p->b->core.qual > 59 ? 59 : p->b->core.qual;
+ imq *= nqual_over_60;
+
+ int sc_len, slen, epos, sc_end;
+
+ // Only need to gather stats on one type, as it's
+ // identical calculation for all the subsequent ones
+ // and we're sharing the same stats array
+ if (t == 0) {
+ // Gather stats for INFO field to aid filtering.
+ // mq and sc_len not very helpful for filtering, but could
+ // help in assigning a better QUAL value.
+ //
+ // Pos is slightly useful.
+ // Base qual can be useful, but need qual prior to BAQ?
+ // May need to cache orig quals in aux tag so we can fetch
+ // them even after mpileup step.
+ get_pos(bca, p, &sc_len, &slen, &epos, &sc_end);
+
+ assert(imq >= 0 && imq < bca->nqual);
+ assert(epos >= 0 && epos < bca->npos);
+ assert(sc_len >= 0 && sc_len < 100);
+ if (p->indel) {
+ bca->ialt_mq[imq]++;
+ bca->ialt_scl[sc_len]++;
+ bca->ialt_pos[epos]++;
+ } else {
+ bca->iref_mq[imq]++;
+ bca->iref_scl[sc_len]++;
+ bca->iref_pos[epos]++;
+ }
+ }
+
+ int qbeg, qpos, qend, tbeg, tend, kk;
+ uint8_t *seq = bam_get_seq(p->b);
+ uint32_t *cigar = bam_get_cigar(p->b);
+ if (p->b->core.flag & BAM_FUNMAP) continue;
+
+ // FIXME: the following loop should be better moved outside;
+ // nonetheless, realignment should be much slower anyway.
+ for (kk = 0; kk < p->b->core.n_cigar; ++kk)
+ if ((cigar[kk]&BAM_CIGAR_MASK) == BAM_CREF_SKIP)
+ break;
+ if (kk < p->b->core.n_cigar)
+ continue;
+
+ // determine the start and end of sequences for alignment
+ int left2 = left, right2 = right;
+ int min_win_size = MAX(-biggest_del, biggest_ins);
+ min_win_size += ABS(left_shift) + ABS(right_shift);
+ {
+ rep_ele *reps, *elt, *tmp;
+ reps = find_STR(tcons[0], tcon_len[0], 0);
+ //int max_str = 0;
+ int tot_str = 0;
+ DL_FOREACH_SAFE(reps, elt, tmp) {
+ // if (max_str < elt->end - elt->start)
+ // max_str = elt->end - elt->start;
+ tot_str += elt->end - elt->start;
+ DL_DELETE(reps, elt);
+ free(elt);
+ }
+
+ // Ideally max_str should be enough, but it's still not
+ // sufficient in longer range some repeats.
+ //min_win_size += max_str;
+ min_win_size += tot_str;
+ }
+ min_win_size += 10;
+
+// TEST 8
+ if (p->b->core.l_qseq > 1000) {
+ // long read data needs less context. It also tends to
+ // have many more candidate indels to investigate so
+ // speed here matters more.
+ if (pos - left >= min_win_size)
+ left2 = MAX(left2, pos - min_win_size);
+ if (right-pos >= min_win_size)
+ right2 = MIN(right2, pos + min_win_size);
+ }
+
+ // Genomic coords for first and last base of query
+ // alignment. This is only used in bcf_cgp_align_score
+ // for computing scores by looking for the proximity
+ // of STRs with the end of the query alignment.
+ int r_start = p->b->core.pos;
+ int r_end = bam_cigar2rlen(p->b->core.n_cigar,
+ bam_get_cigar(p->b));
+ r_end += -1 + r_start;
+
+
+ // Map left2/right2 genomic coordinates to qbeg/qend
+ // query coordinates. The query may not span the
+ // entire left/right region, so this also returns the
+ // equivalent genomic coords for qbeg/qend in tbeg/tend.
+ qbeg = tpos2qpos(&p->b->core, bam_get_cigar(p->b),
+ left2, 0, &tbeg);
+ qpos = tpos2qpos(&p->b->core, bam_get_cigar(p->b), pos,
+ 0, &tend) - qbeg;
+ qend = tpos2qpos(&p->b->core, bam_get_cigar(p->b),
+ right2, 1, &tend);
+
+ int old_tend = tend;
+ int old_tbeg = tbeg;
+
+ // write the query sequence
+ for (l = qbeg; l < qend; ++l)
+ query[l - qbeg] = seq_nt16_int[bam_seqi(seq, l)];
+
+ // tbeg and tend are the genomic locations equivalent
+ // to qbeg and qend on the sequence.
+ // These may being entirely within our left/right
+ // coordinates over which we've computed the
+ // consensus, or overlapping to left/right.
+ //
+ // We know an estimation of band, plus biggest indel,
+ // so we can trim tbeg/tend to a smaller region if we
+ // wish here. This speeds up BAQ scoring.
+ int wband = band + MAX(-biggest_del, biggest_ins)*2 + 20;
+ int tend1 = left + tcon_len[0] - (left2-left);
+ int tend2 = left + tcon_len[1] - (left2-left);
+ tend1 = MIN(tend1, old_tend + wband);
+ tend2 = MIN(tend2, old_tend + wband);
+ tbeg = MAX(left2, old_tbeg - wband);
+
+ // do realignment; this is the bottleneck.
+ //
+ // Note low score = good, high score = bad.
+ if (tend1 > tbeg && tend2 > tbeg) {
+ //fprintf(stderr, "Num %d\n", i);
+ if (bcf_cgp_align_score(p, bca, types[t], band,
+ (uint8_t *)tcons[0] + left2-left,
+ (uint8_t *)tcons[1] + left2-left,
+ (uint8_t *)query,
+ r_start, r_end,
+ tbeg, tend1, tend2,
+ left2, left + tcon_len[0],
+ qbeg, qend, pos,qpos, -biggest_del,
+ qavg, bca->del_bias,
+ &score[K*n_types + t],
+ &str_len1, &str_len2) < 0) {
+ goto err;
+ }
+#ifdef ALIGN_DEBUG
+ fprintf(stderr, "type %d %x / %x\t%s\n",
+ types[t],
+ score[K*n_types + t] >> 8,
+ score[K*n_types + t] & 0xff,
+ bam_get_qname(p->b));
+#endif
+ } else {
+ // place holder large cost for reads that cover the
+ // region entirely within a deletion (thus tend < tbeg).
+ score[K*n_types + t] = 0xffffff;
+ }
+ }
+ free(tcons);
+ }
+ }
+
+ // compute indelQ
+ if (!(l_run_base & l_run_ins))
+ l_run = 1; // different base type in ins to flanking region.
+ n_alt = bcf_cgp_compute_indelQ(n, n_plp, plp, bca, inscns, l_run, max_ins,
+ ref_type, types, n_types, qavg, score,
+ str_len1, str_len2);
+
+ err:
+ // free
+ free(query);
+ free(score);
+ free(types);
+ free(inscns);
+
+ return n_alt > 0? 0 : -1;
+}
--- /dev/null
+#include "bcftools.pysam.h"
+
+/* bam2bcf_indel.c -- indel caller.
+
+ Copyright (C) 2010, 2011 Broad Institute.
+ Copyright (C) 2012-2014,2016-2017, 2021-2024 Genome Research Ltd.
+
+ Author: Heng Li <lh3@sanger.ac.uk>
+ Petr Danecek <pd3@sanger.ac.uk>
+ James Bonfield <jkb@sanger.ac.uk>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE. */
+
+// Show consensus
+//#define CONS_DEBUG
+
+// Show alignments to consensus
+//#define ALIGN_DEBUG
+
+#include <assert.h>
+#include <ctype.h>
+#include <string.h>
+#include <math.h>
+#include <htslib/hts.h>
+#include <htslib/sam.h>
+#include <htslib/khash_str2int.h>
+#include "bam2bcf.h"
+#include "str_finder.h"
+
+#include <htslib/ksort.h>
+// Is there no way to share these between the 3 implementations?
+KSORT_INIT_STATIC_GENERIC(uint32_t)
+
+#define MINUS_CONST 0x10000000
+
+#define MAX_TYPES 64
+
+#ifndef MIN
+# define MIN(a,b) ((a)<(b)?(a):(b))
+#endif
+
+#ifndef ABS
+# define ABS(a) ((a)<0?-(a):(a))
+#endif
+
+#ifndef MAX
+# define MAX(a,b) ((a)>(b)?(a):(b))
+#endif
+
+// l is the relative gap length and l_run is the length of the homopolymer
+// on the reference.
+//
+// Larger seqQ is good, so increasing tandemQ calls more indels,
+// and longer l_run means fewer calls. It is capped later at 255.
+// For short l_runs, the qual is simply based on size of indel
+// larger ones being considered more likely to be real.
+// Longer indels get assigned a score based on the relative indel size
+// to homopolymer, where l_run base will have already been verified by
+// the caller to ensure it's compatible.
+static inline int est_seqQ(const bcf_callaux_t *bca, int l, int l_run, int str_len)
+{
+ int q, qh;
+ // Short indels are more likely sequencing error than large ones.
+ // So "seqQ" scales with size of observation "l".
+ //
+ // Note openQ and extQ are error likelihoods in Phred scale. Hence high
+ // openQ means we're very unlikely to miscall an indel.
+ // Ie it's not the open/ext "costs" normally used in alignment; more the reverse.
+ //
+ // We use MIN(q,qh) below, so we can remove the q component by specifying
+ // a large -o parameter in mpileup.
+ q = bca->openQ + bca->extQ * (abs(l) - 1);
+
+ // Orig method; best with Illumina (high openQ)
+// qh = bca->tandemQ * (double)abs(l) / l_run + .499;
+
+ // Penalise longer homopolymers quadratically more, but boost shorter ones.
+ // Best with CCS (low openQ)
+ //qh = 2 * bca->tandemQ * pow((double)abs(l) / l_run, 1.5) + .499;
+
+ // (l/l_run)^1.26 for openQ=25 or ^1 for openQ=40.
+// double openQ = MIN(40, bca->openQ);
+// qh = (30/openQ) * bca->tandemQ
+// * pow((double)abs(l) / l_run, 1/sqrt(openQ/40)) + .499;
+
+ // Linear scaled on openQ too
+ qh = bca->tandemQ * (double)abs(l) / l_run + .499;
+
+ // Generic maybe ?
+ // power = 1/sqrt(MIN(40,bca->openQ)/40.);
+ // qh = ... * pow((double)abs(l)/l_run, power)
+
+ // bam2bcf.c caps has "if q>seqQ) q=seqQ" so it caps base qual 'q'.
+ // A 1bp indel would therefore have a maximum qual it could be considered based
+ // on open+ext. Hence why openQ is phred score indicating if the base is real
+ // or an over/under-call. (high openQ means high trust in base)
+ return q < qh? q : qh;
+}
+
+// Part of bcf_call_gap_prep.
+//
+// Scans the pileup to identify all the different sizes of indels
+// present.
+// types[] returned is sorted by size, from smallest (maybe negative) to largest.
+//
+// Returns types and fills out n_types_r, max_rd_len_r and ref_type_r,
+// or NULL on error.
+static int *bcf_cgp_find_types(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, bcf_callaux_t *bca, const char *ref,
+ int *max_rd_len_r, int *n_types_r,
+ int *ref_type_r, int *N_r) {
+ int i, j, t, s, N, m, max_rd_len, n_types;
+ int n_alt = 0, n_tot = 0, indel_support_ok = 0;
+ uint32_t *aux;
+ int *types;
+
+ // N is the total number of reads
+ for (s = N = 0; s < n; ++s)
+ N += n_plp[s];
+
+ bca->max_support = bca->max_frac = 0;
+ aux = (uint32_t*) calloc(N + 1, 4);
+ if (!aux)
+ return NULL;
+
+ m = max_rd_len = 0;
+ aux[m++] = MINUS_CONST; // zero indel is always a type (REF)
+
+ // Fill out aux[] array with all the non-zero indel sizes.
+ // Also tally number with indels (n_alt) and total (n_tot).
+ for (s = 0; s < n; ++s) {
+ int na = 0, nt = 0;
+ for (i = 0; i < n_plp[s]; ++i) {
+ const bam_pileup1_t *p = plp[s] + i;
+ ++nt;
+ if (p->indel != 0) {
+ ++na;
+ aux[m++] = MINUS_CONST + p->indel;
+ }
+
+ // FIXME: cache me in pileup struct.
+ j = bam_cigar2qlen(p->b->core.n_cigar, bam_get_cigar(p->b));
+ if (j > max_rd_len) max_rd_len = j;
+ }
+ double frac = (double)na/nt;
+ if ( !indel_support_ok && na >= bca->min_support
+ && frac >= bca->min_frac )
+ indel_support_ok = 1;
+ if ( na > bca->max_support && frac > 0 )
+ bca->max_support = na, bca->max_frac = frac;
+
+ n_alt += na;
+ n_tot += nt;
+ }
+
+ // Sort aux[] and dedup
+ ks_introsort(uint32_t, m, aux);
+ for (i = 1, n_types = 1; i < m; ++i)
+ if (aux[i] != aux[i-1]) ++n_types;
+
+ // Taking totals makes it hard to call rare indels (IMF filter)
+ if ( !bca->per_sample_flt )
+ indel_support_ok = ( (double)n_alt / n_tot < bca->min_frac
+ || n_alt < bca->min_support )
+ ? 0 : 1;
+ if ( n_types == 1 || !indel_support_ok ) { // then skip
+ free(aux);
+ return NULL;
+ }
+
+ // Bail out if we have far too many types of indel
+ if (n_types >= MAX_TYPES) {
+ free(aux);
+ // TODO revisit how/whether to control printing this warning
+ if (hts_verbose >= 2)
+ fprintf(bcftools_stderr, "[%s] excessive INDEL alleles at position %d. "
+ "Skip the position.\n", __func__, pos + 1);
+ return NULL;
+ }
+
+ // To prevent long stretches of N's to be mistaken for indels
+ // (sometimes thousands of bases), check the number of N's in the
+ // sequence and skip places where half or more reference bases are Ns.
+ int nN=0, i_end = pos + (2*bca->indel_win_size < max_rd_len
+ ?2*bca->indel_win_size : max_rd_len);
+ for (i=pos; i<i_end && ref[i]; i++)
+ nN += ref[i] == 'N';
+ if ( nN*2>(i-pos) ) {
+ free(aux);
+ return NULL;
+ }
+
+ // Finally fill out the types[] array detailing the size of insertion
+ // or deletion.
+ types = (int*)calloc(n_types, sizeof(int));
+ if (!types) {
+ free(aux);
+ return NULL;
+ }
+ t = 0;
+ for (i = 0; i < m; ++i) {
+ int sz = (int32_t)(aux[i] - MINUS_CONST);
+ int j;
+ for (j = i+1; j < m; j++)
+ if (aux[j] != aux[i])
+ break;
+
+ if (sz == 0
+ || (j-i >= bca->min_support &&
+ // Note, doesn't handle bca->per_sample_flt yet
+ (bca->per_sample_flt
+ || (double)(j-i) / n_tot >= bca->min_frac)))
+ types[t++] = sz;
+ i = j-1;
+ }
+ free(aux);
+
+ if (t <= 1) {
+ free(types);
+ return NULL;
+ }
+ n_types = t;
+
+ // Find reference type; types[?] == 0)
+ for (t = 0; t < n_types; ++t)
+ if (types[t] == 0) break;
+
+ *ref_type_r = t;
+ *n_types_r = n_types;
+ *max_rd_len_r = max_rd_len;
+ *N_r = N;
+
+ return types;
+}
+
+// Increment ins["str"] and freq["str"]
+#define NI 100 // number of alternative insertion sequences
+// Could use a hash table too, but expectation is a tiny number of alternatives
+typedef struct {
+ char *str[NI];
+ int len[NI];
+ int freq[NI];
+} str_freq;
+
+static int bcf_cgp_append_cons(str_freq *sf, char *str, int len, int freq) {
+ int j;
+
+ for (j = 0; j < NI && sf->str[j]; j++) {
+ if (sf->len[j] == len && memcmp(sf->str[j], str, len) == 0)
+ break;
+ }
+ if (j >= NI)
+ return 0; // too many choices; discard
+
+ sf->freq[j]+=freq;
+ if (!sf->str[j]) {
+ // new insertion
+ if (!(sf->str[j] = malloc(len+1)))
+ return -1;
+ memcpy(sf->str[j], str, len);
+ sf->len[j] = len;
+ }
+
+ return 0;
+}
+
+/*
+ * Compute the consensus for a specific indel type at pos.
+ *
+ * left_shift is the number of inserted(+) or deleted(-) bases added to
+ * the consensus before we get to pos. This is necessary so the alignment
+ * band is correct as it's expected to start at left/right edges in
+ * sync
+ *
+ * We accumulate into several buffers for counting base types:
+ * cons_base - consensus of data with p->indel == type, bases or gap
+ * ref_base - consensus of data with p->indel != type, bases or gap
+ * cons_ins - consensus of data with p->indel == type, insertions
+ * ref_ins - consensus of data with p->indel == type, bases or gap
+ *
+ * The purpose of cons_ins vs cons_base is if we have very low
+ * coverage due to nearly all reads being another type, then we can
+ * still get a robust consensus using the other data. If we don't
+ * have shallow data, then we'll not use as much of ref_base as we may
+ * have correlated variants.
+ *
+ * Eg:
+ * REF: AGCTATGAGGCTGATA
+ * SEQ: AGGTAGGAGGGTGATA (x1)
+ * SEQ: AGCTACGAGG*TGATA (x24)
+ * SEQ: AGCTACTAGG*TGATA (x24)
+ *
+ * Cons for no-del is Cs not Gs. Cannot trust it, so use N if shallow.
+ * CON: AGCTACNAGGGTGATA
+ *
+ * There are still some problems in cons_ins vs ref_ins assignment.
+ * We sometimes seem multiple similar-length insertions added at
+ * different locations. Ideally we'd like to consider these as all
+ * the same insertion if the size is the same and it's comparable seq.
+ */
+#define MAX_INS 8192
+static char **bcf_cgp_consensus(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, bcf_callaux_t *bca, const char *ref,
+ int ref_len, int left, int right,
+ int sample, int type, int biggest_del,
+ int *left_shift, int *right_shift,
+ int *band, int *tcon_len, int *cpos_pos,
+ int pos_l, int pos_r) {
+ // Map ASCII ACGTN* to 012345
+ static uint8_t base6[256] = {
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,5,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ //A C G *^ T
+ 4,0,4,1,4,4,4,2, 4,4,4,4,4,4,4,4, 4,4,4,4,3,3,4,4, 4,4,4,4,4,4,4,4,
+ 4,0,4,1,4,4,4,2, 4,4,4,4,4,4,4,4, 4,4,4,4,3,3,4,4, 4,4,4,4,4,4,4,4,
+
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,
+ };
+
+ // single base or del
+ int (*cons_base)[6] = calloc(right - left + 1, sizeof(*cons_base));
+ // multi-base insertions
+ str_freq *cons_ins = calloc(right - left + 1, sizeof(*cons_ins));
+
+ // non-indel ref for all reads on this sample, rather than those just
+ // matching type. We use this for handling the case where we have a
+ // homozygous deletion being studied, but with 1 or 2 reads misaligned
+ // and containing a base there.
+ //
+ // Eg if the type[]=0 consensus is made up of a very small sample size,
+ // which is also enriched for highly error prone data. We can use
+ // the other reads from type[] != 0 to flesh out the consensus and
+ // improve accuracy.
+ int (*ref_base)[6] = calloc(right - left + 1, sizeof(*ref_base));
+ str_freq *ref_ins = calloc(right - left + 1, sizeof(*ref_ins));
+ int i, j, k, s = sample;
+ char **cons = NULL;
+
+ if (!cons_base || !cons_ins || !ref_base || !ref_ins)
+ goto err;
+
+ //--------------------------------------------------
+ // Accumulate sequences into cons_base and cons_ins arrays
+ int local_band_max = 0; // maximum absolute deviation from diagonal
+ int total_span_str = 0;
+ int type_depth = 0;
+ for (i = 0; i < n_plp[s]; i++) {
+ const bam_pileup1_t *p = plp[s] + i;
+ bam1_t *b = p->b;
+ int x = b->core.pos; // ref coordinate
+ int y = 0; // seq coordinate
+ uint32_t *cigar = bam_get_cigar(b);
+ uint8_t *seq = bam_get_seq(b);
+
+ int local_band = 0; // current deviation from diagonal
+ for (k = 0; k < b->core.n_cigar; ++k) {
+ int op = cigar[k] & BAM_CIGAR_MASK;
+ int len = cigar[k] >> BAM_CIGAR_SHIFT;
+ int base;
+ int skip_to = 0;
+
+ switch(op) {
+ case BAM_CSOFT_CLIP:
+ y += len;
+ break;
+
+ case BAM_CMATCH:
+ case BAM_CEQUAL:
+ case BAM_CDIFF: {
+ // Can short-cut this with j_start and j_end based on
+ // x+len and left,right
+ for (j = 0; j < len; j++, x++, y++) {
+ if (x < left) continue;
+ if (x >= right) break;
+
+ base = bam_seqi(seq, y);
+ if (p->indel == type)
+ // Convert 4-bit base ambig code to 0,1,2,3,4 range
+ cons_base[x-left][seq_nt16_int[base]]++;
+ else if (x != pos+1) // indel being assessed question
+ ref_base[x-left][seq_nt16_int[base]]++;
+ }
+ break;
+ }
+
+ case BAM_CINS: {
+ if (x >= left && x < right) {
+ local_band += p->indel;
+ if (local_band_max < local_band)
+ local_band_max = local_band;
+ }
+
+ char ins[MAX_INS];
+ for (j = 0; j < len; j++, y++) {
+ if (x < left) continue;
+ if (x >= right)
+ break;
+ base = bam_seqi(seq, y);
+ if (j < MAX_INS)
+ ins[j] = seq_nt16_int[base];
+ }
+
+ // Insertions come before a ref match.
+ // 5I 5M is IIIIIM M M M M events, not
+ // {IIIII,M} M M M M choice. So we need to include the
+ // next match in our sequence when choosing the consensus.
+ if (x >= left && x < right) {
+ int ilen = j<MAX_INS?j:MAX_INS;
+ if (p->indel == type /*&& x == pos+1*/) {
+ // Assume any ins of the same size is the same ins.
+ // (This rescues misaligned insertions.)
+ if (bcf_cgp_append_cons(&cons_ins[x-left], ins,
+ ilen, 1) < 0)
+ goto err;
+ type_depth += (x == pos+1);
+ } else if (x != pos+1){
+ if (bcf_cgp_append_cons(&ref_ins[x-left], ins,
+ ilen, 1) < 0)
+ goto err;
+ }
+ }
+ break;
+ }
+
+ case BAM_CDEL:
+ if (x >= left && x < right) {
+ local_band += p->indel;
+ if (local_band_max < -local_band)
+ local_band_max = -local_band;
+ }
+
+ // Maybe not perfect for I/D combos, but likely sufficient.
+ for (j = 0; j < len; j++, x++) {
+ if (x < left) continue;
+ if (x >= right) break;
+ if ((p->indel == type && !p->is_del) || // starts here
+ (p->indel == 0 && p->is_del && len == -type)) { // left
+ cons_base[x-left][5]++;
+ type_depth += (x == pos+1);
+ } else if (x+len <= pos+1 || (skip_to && x > skip_to))
+ ref_base[x-left][5]++;
+ else if (x <= pos && x+len > pos+1) {
+ // we have a deletion which overlaps pos, but
+ // isn't the same "type". We don't wish to
+ // include these as they may bias the
+ // evaluation by confirming against a
+ // secondary consensus produced with the other
+ // deletion. We set a marker for how long to
+ // skip adding to ref_base.
+ if (x > skip_to)
+ skip_to = x+len;
+ }
+ }
+ break;
+ }
+ }
+
+ if (b->core.pos <= pos_l && x >= pos_r)
+ total_span_str++;
+
+ // Also track the biggest deviation +/- from diagonal. We use
+ // this band observation in our BAQ alignment step.
+ if (*band < local_band_max)
+ *band = local_band_max;
+ }
+
+ //--------------------------------------------------
+ // Expand cons_base to include depth from ref_base/ref_ins
+ // Caveat: except at pos itself, where true ref is used if type != 0
+
+#if 1 // TEST 1
+ // We could retest this heuristic further maybe.
+ for (i = 0; i < right-left; i++) {
+ // Total observed depth
+ int t = cons_base[i][0] + cons_base[i][1] + cons_base[i][2] +
+ cons_base[i][3] + cons_base[i][4] + cons_base[i][5];
+ for (j = 0; j < NI; j++) {
+ if (!cons_ins[i].str[j])
+ break;
+ t += cons_ins[i].freq[j];
+ }
+
+ // Similarly for depth on the non-ALT calls (NB: not necessarily
+ // REF as maybe it's other ALTs).
+ int r = ref_base[i][0] + ref_base[i][1] + ref_base[i][2] +
+ ref_base[i][3] + ref_base[i][4] + ref_base[i][5];
+ for (j = 0; j < NI; j++) {
+ if (!ref_ins[i].str[j])
+ break;
+ r += ref_ins[i].freq[j];
+ }
+
+ // When evaluating this particular indel, we don't want to
+ // penalise alignments by SNP errors elsewhere. This can
+ // happen when we have low depth for a particular 'type'.
+ //
+ // So add in a little data from ref_base/ref_ins.
+ double rfract = (r - t*2)*.75 / (r+1);
+
+ if (rfract < 1.01 / (r+1e-10))
+ rfract = 1.01 / (r+1e-10); // low depth compensation
+// if (rfract > 0.2)
+// rfract = 0.2;
+
+ // TODO: consider limiting rfract so we never drown out the
+ // signal. We want to use the remaining data only to correct
+ // for sequencing errors in low depth alleles. If we get
+ // conflicts, it's better to use N than to change a base
+ // incase that variant is genuine.
+ if (i+left >= pos+1 && i+left < pos+1-biggest_del) {
+ // We're overlapping the current indel region, so
+ // we don't wish to bring in evidence from the other
+ // "type" data as it'll harm calling.
+ continue;
+ } else {
+ // Otherwise add in a portion of other data to
+ // boost low population numbers.
+ cons_base[i][0] += rfract * ref_base[i][0];
+ cons_base[i][1] += rfract * ref_base[i][1];
+ cons_base[i][2] += rfract * ref_base[i][2];
+ cons_base[i][3] += rfract * ref_base[i][3];
+ cons_base[i][4] += rfract * ref_base[i][4];
+ cons_base[i][5] += rfract * ref_base[i][5];
+ }
+
+ // Similarly for insertions too; consider a different rfract here?
+ for (j = 0; j < NI; j++) {
+ if (!ref_ins[i].str[j])
+ break;
+ if (bcf_cgp_append_cons(&cons_ins[i],
+ ref_ins[i].str[j], ref_ins[i].len[j],
+ rfract * ref_ins[i].freq[j]) < 0)
+ goto err;
+ }
+ }
+#endif
+
+ //--------------------------------------------------
+ // Allocate consensus buffer, to worst case length
+ int max_len = right-left;
+ for (i = 0; i < right-left; i++) {
+ if (!cons_ins[i].str[0])
+ continue;
+
+ int ins = 0;
+ for (j = 0; j < NI; j++) {
+ if (!cons_ins[i].str[j])
+ break;
+ if (cons_ins[i].str[j] && ins < cons_ins[i].len[j])
+ ins = cons_ins[i].len[j];
+ }
+ max_len += ins;
+ }
+ max_len += MAX(0, type); // incase type inserted bases never occur
+ cons = malloc((max_len+1)*2 + sizeof(char *)*2);
+ if (!cons)
+ goto err;
+ cons[0] = (char *)&cons[2];
+ cons[1] = cons[0] + max_len+1;
+
+ //--------------------------------------------------
+ // Merge insertions where they are the same length but different
+ // sequences.
+ // NB: we could just index by length and have accumulators for each,
+ // instead of storing separately and merging later (here).
+ // Ie str_freq.str is [NI][5] instead.
+ for (i = 0; i < right-left; i++) {
+ int ins[MAX_INS][5];
+ for (j = 0; j < NI; j++) {
+ if (!cons_ins[i].str[j])
+ break;
+
+ if (cons_ins[i].freq[j] == 0)
+ continue; // already merged
+
+ int l;
+ for (l = 0; l < cons_ins[i].len[j]; l++) {
+ // Append to relevant frequency counter, zero all others
+ ins[l][0] = ins[l][1] = ins[l][2] = ins[l][3] = ins[l][4] = 0;
+ uint8_t b = cons_ins[i].str[j][l];
+ ins[l][b] = cons_ins[i].freq[j];
+ }
+
+ // Merge other insertions of the same length to ins[] counters
+ for (k = j+1; k < NI; k++) {
+ if (!cons_ins[i].str[k])
+ break;
+ if (cons_ins[i].len[k] != cons_ins[i].len[j])
+ continue;
+ if (cons_ins[i].freq[k] == 0)
+ continue; // redundant?
+
+ // Merge str[j] and str[k]
+ for (l = 0; l < cons_ins[i].len[k]; l++) {
+ uint8_t b = cons_ins[i].str[k][l];
+ ins[l][b] += cons_ins[i].freq[k];
+ }
+ cons_ins[i].freq[j] += cons_ins[i].freq[k];
+ cons_ins[i].freq[k] = 0;
+ }
+
+ // Now replace ins[j] with the consensus insertion of this len.
+ for (l = 0; l < cons_ins[i].len[j]; l++) {
+ int max_v = 0, base = 0;
+ int tot = ins[l][0] + ins[l][1] + ins[l][2]
+ + ins[l][3] + ins[l][4];
+ if (max_v < ins[l][0]) max_v = ins[l][0], base = 0;
+ if (max_v < ins[l][1]) max_v = ins[l][1], base = 1;
+ if (max_v < ins[l][2]) max_v = ins[l][2], base = 2;
+ if (max_v < ins[l][3]) max_v = ins[l][3], base = 3;
+ if (max_v < ins[l][4]) max_v = ins[l][4], base = 4;
+
+ cons_ins[i].str[j][l] = (max_v > 0.6*tot) ? base : 4;
+ }
+ }
+ }
+
+#define CONS_CUTOFF .40 // % needed for base vs N
+#define CONS_CUTOFF_DEL .35 // % to include any het del
+#define CONS_CUTOFF2 .80 // % needed for gap in cons[1]
+#define CONS_CUTOFF_INC .35 // % to include any insertion cons[0]
+#define CONS_CUTOFF_INC2 .80 // % to include any insertion cons[1] HOM
+#define CONS_CUTOFF_INS .60 // and then 60% needed for it to be bases vs N
+
+ //--------------------------------------------------
+ // Walk through the frequency arrays to call the consensus.
+ // We produce cons[0] and cons[1]. Both include strongly
+ // homozygous indels. Both also include the indel at 'pos'.
+ // However for heterozygous indels we call the most likely event
+ // for cons[0] and the less-likely alternative in cons[1].
+ // TODO: a proper phase analysis so multiple events end up
+ // combining together into the correct consensus.
+ *left_shift = 0;
+ *right_shift = 0;
+ int cnum;
+
+ // Het call filled out in cnum==0 (+ve or -ve).
+ // Used in cnum==1 to do the opposite of whichever way we did before.
+ int heti[MAX_INS] = {0}, hetd[MAX_INS] = {0};
+
+ *cpos_pos = -1;
+ for (cnum = 0; cnum < 2; cnum++) {
+ for (i = k = 0; i < right-left; i++) {
+ // Location in consensus matching the indel itself
+ if (i >= pos-left+1 && *cpos_pos == -1)
+ *cpos_pos = k;
+
+ int max_v = 0, max_v2 = 0, max_j = 4, max_j2 = 4, tot = 0;
+ for (j = 0; j < 6; j++) {
+ // Top 2 consensus calls
+ if (max_v < cons_base[i][j]) {
+ max_v2 = max_v, max_j2 = max_j;
+ max_v = cons_base[i][j], max_j = j;
+ } else if (max_v2 < cons_base[i][j]) {
+ max_v2 = cons_base[i][j], max_j2 = j;
+ }
+ tot += cons_base[i][j];
+ }
+
+ // +INS
+ int max_v_ins = 0, max_j_ins = 0;
+ int tot_ins = 0;
+ for (j = 0; j < NI; j++) {
+ if (i+left==pos+1)
+ if (type > 0 && i+left == pos+1
+ && cons_ins[i].len[j] < type && j == 0) {
+ cons_ins[i].str[j] = realloc(cons_ins[i].str[j], type);
+ if (!cons_ins[i].str[j])
+ goto err;
+ memset(cons_ins[i].str[j] + cons_ins[i].len[j],
+ 4, type - cons_ins[i].len[j]);
+ cons_ins[i].len[j] = type;
+ }
+ if (!cons_ins[i].str[j])
+ break;
+ if (cons_ins[i].freq[j] == 0)
+ continue; // previously merged
+
+ if (max_v_ins < cons_ins[i].freq[j])
+ //if (i != pos-left+1 || cons_ins[i].len[j] == type)
+ max_v_ins = cons_ins[i].freq[j], max_j_ins = j;
+ tot_ins += cons_ins[i].freq[j];
+ }
+
+ // NB: tot is based on next matching base, so it includes
+ // everything with or without the insertion.
+ int tot_sum = tot;
+ int always_ins =
+ (i == pos-left+1 && type>0) || // current eval
+ max_v_ins > CONS_CUTOFF_INC2*tot_sum;// HOM
+ int het_ins = 0;
+ if (!always_ins && max_v_ins >= bca->min_support) {
+ // Candidate HET ins.
+ if (cnum == 0) {
+ het_ins = max_v_ins > CONS_CUTOFF_INC * tot_sum;
+ if (i < MAX_INS) heti[i] = het_ins
+ ? 1
+ : (max_v_ins > .3*tot_sum ? -1:0);
+ } else {
+ // HET but uncalled before
+ het_ins = i < MAX_INS ? (heti[i] == -1) : 0;
+ }
+ }
+
+ if (always_ins || het_ins) {
+ if (max_v_ins > CONS_CUTOFF_INS*tot_ins) {
+ // Insert bases
+ for (j = 0; j < cons_ins[i].len[max_j_ins]; j++) {
+ if (cnum == 0) {
+ if (k < pos-left+*left_shift)
+ (*left_shift)++;
+ else
+ (*right_shift)++;
+ }
+ cons[cnum][k++] = cons_ins[i].str[max_j_ins][j];
+ }
+ } else {
+ for (j = 0; j < cons_ins[i].len[max_j_ins]; j++)
+ cons[cnum][k++] = 4; // 'N';
+ }
+ }
+
+ // Call deletions & bases
+ int always_del = (type < 0 && i > pos-left && i <= pos-left-type)
+ || cons_base[i][5] > CONS_CUTOFF2 * tot; // HOM del
+ int het_del = 0;
+ if (!always_del && cons_base[i][5] >= bca->min_support) {
+ // Candidate HET del.
+ if (cnum == 0) {
+ int tot2 = tot;
+ if (i > pos-left && i <= pos-left-biggest_del)
+ tot2 = total_span_str - type_depth;
+ het_del = cons_base[i][5] >= CONS_CUTOFF_DEL * tot2;
+
+ if (i < MAX_INS) {
+ if (i > pos-left && i <= pos-left-biggest_del)
+ hetd[i] = 0;
+ else
+ hetd[i] = het_del
+ ? 1
+ : (cons_base[i][5] >= .3 * tot2 ? -1 : 0);
+ }
+ } else {
+ // HET del uncalled on cnum 0
+ het_del = i < MAX_INS ? (hetd[i] == -1) : 0;
+ if (max_j == 5 && het_del == 0) {
+ max_v = max_v2;
+ max_j = max_j2;
+ }
+ }
+ }
+ if (always_del || het_del) {
+ // Deletion
+ if (k < pos-left+*left_shift)
+ (*left_shift)--;
+ else
+ (*right_shift)++;
+ } else {
+ // Finally the easy case - a non-indel base or an N
+ if (max_v > CONS_CUTOFF*tot)
+ cons[cnum][k++] = max_j; // "ACGTN*"
+ else if (max_v > 0)
+ cons[cnum][k++] = 4; // 'N';
+ else {
+ cons[cnum][k] = left+k < ref_len
+ ? base6[(uint8_t)ref[left+k]]
+ : 4;
+ k++;
+ }
+ }
+ }
+
+ tcon_len[cnum] = k;
+ }
+
+ // TODO: replace by io_lib's string pool for rapid tidying.
+ // For now this isn't the bottleneck though.
+ for (i = 0; i < right-left; i++) {
+ for (j = 0; j < NI; j++) {
+ if (cons_ins[i].str[j])
+ free(cons_ins[i].str[j]);
+ if (ref_ins[i].str[j])
+ free(ref_ins[i].str[j]);
+ }
+ }
+
+ err:
+ free(cons_base);
+ free(ref_base);
+ free(cons_ins);
+ free(ref_ins);
+
+ return cons;
+}
+
+// A rename of bcf_cgp_calc_cons from bam2bcf_indel.c
+//
+// Compute the insertion consensus for this sample 's' via a basic
+// majority rule.
+//
+// TODO: merge this into bcf_cgp_consensus as another return value?
+static char *bcf_cgp_calc_ins_cons(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, int *types, int n_types,
+ int max_ins, int s) {
+ return bcf_cgp_calc_cons(n, n_plp, plp, pos, types, n_types, max_ins, s);
+}
+
+#define MAX(a,b) ((a)>(b)?(a):(b))
+#define MIN(a,b) ((a)<(b)?(a):(b))
+
+// Compile with LIBS="-L. -ldl -ledlib" CLD=g++
+
+// This is faster than ksw and BAQ, meaning we can use larger --indel-size and
+// get a more accurate context, improving alignments further. This *may*
+// compensate for reduced sensitivity.
+#include "edlib.h"
+int edlib_glocal(uint8_t *ref, int l_ref, uint8_t *query, int l_query,
+ double m, double del_bias)
+{
+ EdlibAlignConfig cfg =
+ edlibNewAlignConfig(
+ //ABS(type)+ABS(l_ref-l_query)+10,
+ -1, // k; use small positive for faster alignment
+ EDLIB_MODE_HW, // mode
+#ifdef ALIGN_DEBUG
+ EDLIB_TASK_PATH,
+#else
+ EDLIB_TASK_LOC,
+#endif
+ NULL, // additionalEqualities
+ 0); // additionalEqualitiesLength
+ EdlibAlignResult r =
+ edlibAlign((char *)query, l_query, (char *)ref, l_ref, cfg);
+
+ if (r.status != EDLIB_STATUS_OK || r.numLocations < 1 ||
+ !r.endLocations || !r.startLocations) {
+ edlibFreeAlignResult(r);
+ return INT_MAX;
+ }
+
+#ifdef ALIGN_DEBUG
+ // NB: Needs linking against the C++ libedlib.a as our cut-down C
+ // implementation misses the alignment generation code.
+ {
+ int i, j = 0, pt = r.startLocations[0], pq = 0;
+ char line1[80];
+ char line2[80];
+ char line3[80];
+ for (i = 0; i < r.alignmentLength && pt < r.endLocations[0]; i++) {
+ int n;
+ switch (n = r.alignment[i]) {
+ case 0: // match
+ case 3: // mismatch
+ line1[j] = "ACGTN"[ref[pt++]];
+ line2[j] = "ACGTN"[query[pq++]];
+ line3[j] = " x"[n==3];
+ break;
+ case 2: // insertion to ref
+ line1[j] = "ACGTN"[ref[pt++]];
+ line2[j] = '-';
+ line3[j] = '-';
+ break;
+ case 1: // insertion to query
+ line1[j] = '-';
+ line2[j] = "ACGTN"[query[pq++]];
+ line3[j] = '+';
+ break;
+ }
+
+ if (++j == sizeof(line1)) {
+ fprintf(bcftools_stderr, "%.*s\n", j, line1);
+ fprintf(bcftools_stderr, "%.*s\n", j, line2);
+ fprintf(bcftools_stderr, "%.*s\n", j, line3);
+ j = 0;
+ }
+ }
+ if (j) {
+ fprintf(bcftools_stderr, "%.*s\n", j, line1);
+ fprintf(bcftools_stderr, "%.*s\n", j, line2);
+ fprintf(bcftools_stderr, "%.*s\n", j, line3);
+ }
+ }
+#endif
+
+ // Aligned target length minus query length is an indication of the number
+ // of insertions and/or deletions.
+ //
+ // For CIGAR 10M1I10M t_len > l_query ("AC" / "ATC")
+ // For CIGAR 10M1D10M t_len < l_query ("ATC" / "AC")
+ // Hence t_len-l_query is -ve for net insertions and +ve for net deletions.
+ // If we compute nins and ndel directly via walking though EDLIB_TASK_PATH
+ // we'll see t_len-l_query == ndel-nins.
+ //
+ // If a technology has a significantly higher chance of making deletion
+ // errors than insertion errors, then we would view deletions as less
+ // indicative of this sequence not coming from this candidate allele than
+ // if it had insertion (as the deletions are more likely to be errors
+ // rather than real, relative to the insertions). Hence we can skew the
+ // score by the net delta of num_del - num_ins.
+ //
+ // Note this is an approximation that doesn't account for multiple
+ // insertions and deletions within the same sequence, but it is much faster
+ // as it doesn't require EDLIB_TASK_PATH to be computed.
+ //
+ // Given editDistance is +1 for every mismatch, insertion and deletion,
+ // provided the t_len-l_query multiplier < 1 then this is always +ve.
+
+ int t_len = *r.endLocations - *r.startLocations + 1;
+ int score = m*(r.editDistance - del_bias*(t_len - l_query));
+
+ edlibFreeAlignResult(r);
+ return score;
+}
+
+// Part of bcf_call_gap_prep.
+//
+// Realign using BAQ to get an alignment score of a single read vs
+// a haplotype consensus. TODO: replace BAQ with something more robust.
+//
+// There are many coordinates, so let's explain them.
+// - left, right, tbeg, tend, r_start and r_end are in aligned reference
+// coordinates.
+// left/right start from pos +/- indel_win_size.
+// r_start/r_end are the BAM first and last mapped coord on the reference.
+// tbeg and tend are the intersection of the two.
+// - qbeg and qend are in BAM sequence coordinates
+// - qpos is in sequence coordinates, relative to qbeg.
+//
+// To see what this means, we have illustrations with coordinates
+// above the seqs in reference space and below the seqs in BAM seq space.
+//
+// Overlap left:
+// tbeg tend
+// r_start left pos r_end right
+// REF :..............|--------------------#------:--------------|...
+// SEQ :..............|--------------------#------|
+// 0 qbeg qpos qend
+//
+// Overlap right:
+// r_start tend
+// left tbeg pos right r_end
+// REF ...|--------------:-----#---------------------|...........:
+// SEQ |-----#---------------------|...........:
+// qbeg qpos qend
+// 0
+//
+// The "-" sequence is the bit passed in.
+// Ie ref2 spans left..right and query spans qbeg..qend.
+// We need to adjust ref2 therefore to tbeg..tend.
+//
+// Fills out score
+// Returns 0 on success,
+// <0 on error
+static int bcf_cgp_align_score(bam_pileup1_t *p, bcf_callaux_t *bca,
+ int type, int band,
+ uint8_t *ref1, uint8_t *ref2, uint8_t *query,
+ int r_start, int r_end,
+ int tbeg, int tend1, int tend2,
+ int left, int right,
+ int qbeg, int qend,
+ int pos, int qpos, int max_deletion,
+ double qavg, double del_bias, int *score,
+ int *str_len1_p, int *str_len2_p) {
+ int atype = abs(type);
+ int l, sc1, sc2;
+
+ // Trim poly_Ns at ends of ref.
+ // This helps to keep len(ref) and len(query) similar, to reduce
+ // band size and reduce the chance of -ve BAQ scores.
+ for (l = 0; l < tend1-tbeg && l < tend2-tbeg; l++)
+ if (ref1[l + tbeg-left] != 4 || ref2[l + tbeg-left] != 4)
+ break;
+ if (l > atype)
+ tbeg += l-atype;
+
+ for (l = tend1-tbeg-1; l >= 0; l--)
+ if (ref1[l + tbeg-left] != 4)
+ break;
+ l = tend1-tbeg-1 - l;
+ if (l > atype)
+ tend1 -= l-atype;
+
+ for (l = tend2-tbeg-1; l >= 0; l--)
+ if (ref2[l + tbeg-left] != 4)
+ break;
+ l = tend2-tbeg-1 - l;
+ if (l > atype) {
+ tend2 -= l-atype;
+ }
+
+ // The bottom 8 bits are length-normalised score while
+ // the top bits are unnormalised.
+ //
+ // Try original cons and new cons and pick best.
+ // This doesn't reduce FN much (infact maybe adds very slightly),
+ // but it does reduce GT errors and is a slight reduction to FP.
+
+ double mm = 30; // a const average qual for now. Could tune
+ sc2 = edlib_glocal(ref2 + tbeg - left, tend2 - tbeg,
+ query, qend - qbeg, mm, del_bias);
+
+ if (tend1 != tend2 ||
+ memcmp((char *)ref1 + tbeg - left, (char *)ref2 + tbeg - left,
+ tend1 - tbeg) != 0)
+ sc1 = edlib_glocal(ref1 + tbeg - left, tend1 - tbeg,
+ query, qend - qbeg, mm, del_bias);
+ else
+ sc1 = INT_MAX; // skip
+
+ // Find the best of the two alignments
+ if (sc1 < 0 && sc2 < 0) {
+ *score = 0xffffff;
+ return 0;
+ }
+ if (sc1 < 0) {
+ // sc2 is already correct
+ } else if (sc2 < 0) {
+ sc2 = sc1;
+ } else {
+ // sc1 and sc2 both pass, so use best
+ if (sc2 > sc1)
+ sc2 = sc1;
+ }
+
+ // Sc is overall alignment score, in top 24 bits (SeqQ). It's based
+ // purely on the scores for the whole alignment.
+ // We also have a separate indel score in bottom 8 bits (IndelQ).
+ // This is a function of all sorts of attributes of the candidate indel
+ // itself, such as STR length and the presence of poor quality bases.
+
+ // Used for adjusting indelQ below. Lower l is more likely to call
+ // (--FN, ++FP). (NB CLI --indel_bias is 1/indel_bias var).
+ // Starts as average score per base, and then adjusted based on seq
+ // complexity / quality.
+
+ l = .5*(100. * sc2 / (qend - qbeg) + .499);
+
+ *score = (sc2<<8) | (int)MIN(255, l * bca->indel_bias * .5);
+
+ return 0;
+}
+
+// Part of bcf_call_gap_prep.
+//
+// Returns n_alt on success
+// -1 on failure
+
+// TODO: almost identical to bam2bcf_indel.c's copy, so we could share
+// the code and add a check on bca->edlib.
+static int bcf_cgp_compute_indelQ(int n, int *n_plp, bam_pileup1_t **plp,
+ bcf_callaux_t *bca, char *inscns,
+ int l_run, int max_ins,
+ int ref_type, int *types, int n_types,
+ double qavg, int *score,
+ int str_len1, int str_len2) {
+ // FIXME: n_types has a maximum; no need to alloc - use a #define?
+ int sc[MAX_TYPES], sumq[MAX_TYPES], s, i, j, t, K, n_alt, tmp;
+ memset(sumq, 0, n_types * sizeof(int));
+ int sum_indelQ1[100] = {0}; // n
+ int sum_indelQ2[100] = {0}; // n
+
+ // Confusing variable naming and bit usage.
+ //
+ // score[] is low 8 bits normalised (by len) alignment score
+ // top 24 bits full alignment score
+ // This gets cast into "sct"; mnemonic score-per-indel-type.
+ //
+ // sc = (score<<6) | type (index to types[] array for indel size)
+ // So sc>>14 = score>>(14-6) = score>>8. Ie full alignment score
+ for (s = K = 0; s < n; ++s) {
+ for (i = 0; i < n_plp[s]; ++i, ++K) {
+ bam_pileup1_t *p = plp[s] + i;
+ // Labelling is confusing here.
+ // sct is short for score.
+ // sc is score + t(type)
+ // Why aren't these variable names reversed?
+ int *sct = &score[K*n_types], seqQ, indelQ1=0, indelQ2=0, indelQ=0;
+ for (t = 0; t < n_types; ++t) sc[t] = sct[t]<<6 | t;
+ for (t = 1; t < n_types; ++t) // insertion sort
+ for (j = t; j > 0 && sc[j] < sc[j-1]; --j)
+ tmp = sc[j], sc[j] = sc[j-1], sc[j-1] = tmp;
+
+#ifdef ALIGN_DEBUG
+ fprintf(bcftools_stderr, "READ %s\tscores ", bam_get_qname(p->b));
+ for (t = 0; t < n_types; ++t) {
+ fprintf(bcftools_stderr, "%+2d/%-3d ", types[sc[t]&0x3f], sc[t]>>14);
+ }
+#endif
+
+ /* errmod_cal() assumes that if the call is wrong, the
+ * likelihoods of other events are equal. This is about
+ * right for substitutions, but is not desired for
+ * indels. To reuse errmod_cal(), I have to make
+ * compromise for multi-allelic indels.
+ */
+ if ((sc[0]&0x3f) == ref_type) {
+ // sc >> 14 is the total score. It's been shifted by 8
+ // from normalised score and 6 from type.
+ // &0x3f is type number
+
+ // Best call is REF. Compare vs best indel
+ indelQ = (sc[1]>>14) - (sc[0]>>14);
+ seqQ = est_seqQ(bca, types[sc[1]&0x3f], l_run, str_len1);
+ } else {
+ // look for the reference type
+ for (t = 0; t < n_types; ++t) {
+ if ((sc[t]&0x3f) == ref_type)
+ break;
+ }
+ indelQ = indelQ1 = (sc[t]>>14) - (sc[0]>>14);
+// fprintf(bcftools_stderr, "IndelQ = %d: %d-%d",
+// indelQ, (sc[t]>>14), (sc[0]>>14));
+
+ // Best call is non-ref, compare vs next best non-ref,
+ // or ref if it's just 2 choices (most common case).
+ for (t = 1; t < n_types; t++)
+ if ((sc[t]&0x3f) == ref_type)
+ continue;
+ else break;
+ if (t == n_types)
+ t--; // it's ref, but it'll do as next best.
+ indelQ2 = (sc[t]>>14) - (sc[0]>>14);
+ seqQ = est_seqQ(bca, types[sc[0]&0x3f], l_run, str_len1);
+
+#if 1 // TEST 3
+ indelQ = bca->vs_ref*indelQ1 + (1-bca->vs_ref)*indelQ2;
+#endif
+ }
+
+ // So we lower qual in some, but raise the average to keep FN/FP
+ // ratios up.
+ // Is this key diff for PacBio old vs new HiFi?
+ indelQ /= bca->indel_bias*0.5;
+ indelQ1 /= bca->indel_bias*0.5;
+
+ // Or maybe just *2 if bca->poly_mqual and be done with it?
+ // Or perhaps adjust the MIN(qavg/20, ...) to MIN(qavg/10) ?
+
+ // Skew SeqQ and IndelQ based on a portion of the minimum quality
+ // found within a homopolymer. This is useful where the quality
+ // values are a bit mutable and move around in such data, but less
+ // so on clocked sequencing technologies.
+ //
+ // Enabling this causes lots of GT errors on Illumina.
+ // However on PacBio it's key to removal of false positives.
+ // ONT and UG seem somewhere inbetween.
+ if (bca->poly_mqual) { // TEST 4
+ int qpos = p->qpos, l;
+ uint8_t *seq = bam_get_seq(p->b);
+ uint8_t *qual = bam_get_qual(p->b);
+ int min_q = qual[qpos];
+
+ // scan homopolymer left
+ char baseL = bam_seqi(seq, qpos+1 < p->b->core.l_qseq
+ ? qpos+1 : qpos);
+ for (l = qpos; l >= 0; l--) {
+ if (bam_seqi(seq, l) != baseL)
+ break;
+ if (min_q > qual[l])
+ min_q = qual[l];
+ }
+
+ // scan homo-polymer right (including site of indel)
+ char base = bam_seqi(seq, qpos+1);
+ for (l = qpos+1; l < p->b->core.l_qseq; l++) {
+ if (min_q > qual[l])
+ min_q = qual[l];
+ if (bam_seqi(seq, l) != base)
+ break;
+ }
+
+ // We reduce -h so homopolymers get reduced likelihood of being
+ // called, but then optionally increase or decrease from there
+ // based on base quality. Hence lack of low quality bases in
+ // homopolymer will rescue the score back again, reducing FNs.
+
+ // The score factors here may also be machine specific, but for
+ // now these work well (tuned on PB HiFi).
+ seqQ += MIN(qavg/20, min_q - qavg/10);
+ indelQ += MIN(qavg/20, min_q - qavg/5);
+ indelQ1+= MIN(qavg/20, min_q - qavg/5);
+
+ if (seqQ < 0) seqQ = 0;
+ if (indelQ < 0) indelQ = 0;
+ if (indelQ1< 0) indelQ1= 0;
+ }
+
+ // This is the length-normalised score from bcf_cgp_align_score
+ tmp = sc[0]>>6 & 0xff;
+
+ // reduce indelQ
+ // high score = bad, low score = good; flip for indelQ
+ // low normalised scores leave indelQ unmodified
+ // high normalised scores set indelQ to 0
+ // inbetween scores have a linear scale from indelQ to 0
+// Altering the MAGIC value below (originally 111, but chosen for unknown
+// reasons) is comparable to altering --indel-bias.
+#define TMP_MAGIC 255.0
+
+ indelQ = tmp > TMP_MAGIC? 0 : (int)((1. - tmp/TMP_MAGIC) * indelQ + .499);
+ indelQ1= tmp > TMP_MAGIC? 0 : (int)((1. - tmp/TMP_MAGIC) * indelQ1+ .499);
+
+ indelQ = MIN(indelQ, 255);
+ indelQ1 = MIN(indelQ1, 255);
+
+ // Doesn't really help accuracy, but permits -h to take
+ // affect still.
+ if (indelQ > seqQ) indelQ = seqQ;
+ if (indelQ > 255) indelQ = 255;
+ if (indelQ1> 255) indelQ1= 255;
+ if (seqQ > 255) seqQ = 255;
+
+ // Use 22 bits in total.
+ // 0-7 IndelQ
+ // 8-15 SeqQ
+ // 16-22 Score-per-base
+ p->aux = (sc[0]&0x3f)<<16 | seqQ<<8 | indelQ;
+ sumq[sc[0]&0x3f] += indelQ;
+
+#ifdef ALIGN_DEBUG
+ fprintf(bcftools_stderr, "\t%d\t%d\n", indelQ, seqQ);
+#endif
+
+ // Experiment in p->aux vs sumq.
+ // One gives likelihood of an indel being here, while the other
+ // is likelihood of a specific genotype? But which is which?
+
+ sum_indelQ1[s] += indelQ1;
+ sum_indelQ2[s] += indelQ;
+ }
+ }
+
+ // Determine bca->indel_types[] and bca->inscns.
+ // Sumq[0] is always reference.
+ // Sumq[1] is best non-ref (and maybe better than ref)
+ bca->maxins = max_ins;
+ bca->inscns = (char*) realloc(bca->inscns, bca->maxins * 4);
+ if (bca->maxins && !bca->inscns)
+ return -1;
+ for (t = 0; t < n_types; ++t)
+ sumq[t] = sumq[t]<<6 | t;
+ for (t = 1; t < n_types; ++t) // insertion sort
+ for (j = t; j > 0 && sumq[j] > sumq[j-1]; --j)
+ tmp = sumq[j], sumq[j] = sumq[j-1], sumq[j-1] = tmp;
+ for (t = 0; t < n_types; ++t) // look for the reference type
+ if ((sumq[t]&0x3f) == ref_type) break;
+
+ if (t) { // then move the reference type to the first
+ tmp = sumq[t];
+ for (; t > 0; --t) sumq[t] = sumq[t-1];
+ sumq[0] = tmp;
+ }
+
+ for (t = 0; t < 4; ++t) bca->indel_types[t] = B2B_INDEL_NULL;
+ for (t = 0; t < 4 && t < n_types; ++t) {
+ bca->indel_types[t] = types[sumq[t]&0x3f];
+#ifdef ALIGN_DEBUG
+ fprintf(bcftools_stderr, "TYPE %+2d %d\n", types[t], sumq[t]>>6);
+#endif
+ if (bca->maxins) // potentially an insertion
+ memcpy(&bca->inscns[t * bca->maxins],
+ &inscns[(sumq[t]&0x3f) * max_ins], bca->maxins);
+ }
+
+ // Update p->aux.
+ // If per-alignment type isn't found, then indelQ/seqQ is 0,
+ // otherwise unchanged.
+ for (s = n_alt = 0; s < n; ++s) {
+ for (i = 0; i < n_plp[s]; ++i) {
+ bam_pileup1_t *p = plp[s] + i;
+ int x = types[p->aux>>16&0x3f];
+ for (j = 0; j < 4; ++j)
+ if (x == bca->indel_types[j]) break;
+ p->aux = j<<16 | (j == 4? 0 : (p->aux&0xffff));
+ if ((p->aux>>16&0x3f) > 0) ++n_alt;
+#ifdef ALIGN_DEBUG
+ fprintf(bcftools_stderr, "FIN %s\t%d\t%d\t%d\n",
+ bam_get_qname(p->b), (p->aux>>16)&0x3f,
+ bca->indel_types[(p->aux>>16)&0x3f], p->aux&0xff);
+#endif
+ }
+ }
+
+ return n_alt;
+}
+
+/*
+FIXME: with high number of samples, do we handle IMF correctly? Is it
+fraction of indels across entire data set, or just fraction for this
+specific sample? Needs to check bca->per_sample_flt (--per-sample-mF) opt.
+ */
+
+/*
+ notes:
+ - n .. number of samples
+ - the routine sets bam_pileup1_t.aux of each read as follows:
+ - 6: unused
+ - 6: the call; index to bcf_callaux_t.indel_types .. (aux>>16)&0x3f
+ - 8: estimated sequence quality .. (aux>>8)&0xff
+ - 8: indel quality .. aux&0xff
+ */
+int bcf_edlib_gap_prep(int n, int *n_plp, bam_pileup1_t **plp, int pos,
+ bcf_callaux_t *bca, const char *ref, int ref_len)
+{
+ if (ref == 0 || bca == 0) return -1;
+
+ int i, s, t, n_types, *types = NULL, max_rd_len, left, right, max_ins;
+ int *score = NULL;
+ int N, K, l_run, ref_type, n_alt = -1;
+ char *inscns = NULL, *query = NULL;
+
+ // determine if there is a gap
+ for (s = N = 0; s < n; ++s) {
+ for (i = 0; i < n_plp[s]; ++i)
+ if (plp[s][i].indel != 0) break;
+ if (i < n_plp[s]) break;
+ }
+ if (s == n)
+ // there is no indel at this position.
+ return -1;
+
+ // Find average base quality over this region
+ double qavg = 30, qsum = 0, qcount = 0;
+ int qmax = 0;
+ for (s = 0; s < n; s++) {
+ for (i = 0; i < n_plp[s]; i++) {
+#define QWIN 50
+ bam_pileup1_t *p = plp[s] + i;
+ int kstart = p->qpos - QWIN > 0 ? p->qpos - QWIN : 0;
+ int kend = p->qpos + QWIN < p->b->core.l_qseq
+ ? p->qpos + QWIN : p->b->core.l_qseq;
+ uint8_t *qual = bam_get_qual(p->b);
+ int k;
+ for (k = kstart; k < kend; k++) {
+ qsum += qual[k];
+ qcount++;
+ if (qmax < qual[k])
+ qmax = qual[k];
+ }
+ }
+ }
+ qavg = (qsum+1) / (qcount+1);
+
+ // find out how many types of indels are present
+ types = bcf_cgp_find_types(n, n_plp, plp, pos, bca, ref,
+ &max_rd_len, &n_types, &ref_type, &N);
+ if (!types)
+ goto err;
+
+
+ // calculate left and right boundary, based on type size for a bit more
+ // speed.
+ int max_indel = 20*MAX(ABS(types[0]), ABS(types[n_types-1]))
+ + bca->indel_win_size/4;
+ if (max_indel > bca->indel_win_size)
+ max_indel = bca->indel_win_size;
+ left = pos > max_indel ? pos - max_indel : 0;
+ right = pos + max_indel;
+
+ int del_size = types[0]<0 ? -types[0] : 0;
+ right += del_size;
+
+ // in case the alignments stand out the reference
+ for (i = pos; i < right; ++i)
+ if (ref[i] == 0) break;
+ right = i;
+
+ // compute the likelihood given each type of indel for each read
+ max_ins = types[n_types - 1]; // max_ins is at least 0
+
+ // The length of the homopolymer run around the current position
+ l_run = bcf_cgp_l_run(ref, pos);
+ int l_run_base = seq_nt16_table[(uint8_t)ref[pos+1]];
+ int l_run_ins = 0;
+
+ // construct the consensus sequence (minus indels, which are added later)
+ if (max_ins > 0) {
+ // TODO: replace filling inscns[] with calc_consensus return
+ // so the merges of the insertion consensus for type[t] is
+ // reported directly. (It may need adjustment to avoid N)
+ inscns = bcf_cgp_calc_ins_cons(n, n_plp, plp, pos,
+ types, n_types, max_ins, s);
+ if (!inscns)
+ return -1;
+ }
+
+ query = (char*) calloc(right - left + max_rd_len + max_ins + 2, 1);
+ score = (int*) calloc(N * n_types, sizeof(int));
+ bca->indelreg = 0;
+ double nqual_over_60 = bca->nqual / 60.0;
+
+ int biggest_del = 0;
+ int biggest_ins = 0;
+ for (t = 0; t < n_types; t++) {
+ if (biggest_del > types[t])
+ biggest_del = types[t];
+ if (biggest_ins < types[t])
+ biggest_ins = types[t];
+ }
+ int band = biggest_ins - biggest_del; // NB del is -ve
+
+ // Find left & right extents of STR covering pos, from ref
+ int pos_l = pos, pos_r = pos;
+ {
+ rep_ele *reps, *elt, *tmp;
+ int pstart = MAX(0, pos-30);
+ int pmid = pos-pstart;
+ int pend = MIN(ref_len, pos+30);
+ reps = find_STR((char *)&ref[pstart], pend-pstart, 0);
+ DL_FOREACH_SAFE(reps, elt, tmp) {
+ if (elt->end >= pmid && elt->start <= pmid) {
+ if (pos_l > pstart + elt->start)
+ pos_l = pstart + elt->start;
+ if (pos_r < pstart + elt->end)
+ pos_r = pstart + elt->end;
+ }
+ DL_DELETE(reps, elt);
+ free(elt);
+ }
+ }
+
+ int str_len1 = l_run, str_len2 = l_run/4;
+ for (t = 0; t < n_types; ++t) {
+ int l, ir;
+
+ // Compute indelreg. This is the context in the reference. Eg:
+ //
+ // REF: AG--TTTC Inscns is "TT".
+ // SEQ: AGTTTTTC Indelreg is 3; next 3 "TTT" bases
+ //
+ // => GTTT GTTTTT is call.
+ if (types[t] == 0)
+ ir = 0;
+ else if (types[t] > 0)
+ ir = est_indelreg(pos, ref, types[t], &inscns[t*max_ins]);
+ else
+ ir = est_indelreg(pos, ref, -types[t], 0);
+
+ if (ir > bca->indelreg)
+ bca->indelreg = ir;
+
+ // Realignment score, computed via BAQ
+ for (s = K = 0; s < n; ++s) {
+ char **tcons;
+ int left_shift, right_shift;
+ int tcon_len[2];
+ int cpos_pos;
+ tcons = bcf_cgp_consensus(n, n_plp, plp, pos, bca, ref, ref_len,
+ left, right, s, types[t], biggest_del,
+ &left_shift, &right_shift, &band,
+ tcon_len, &cpos_pos, pos_l, pos_r);
+ // TODO: Consensus for a deletion shouldn't match the
+ // consensus for type 0. Eg consider
+ // vv vv
+ // REF: AATGTGTGAACAA REF: AATGTG--AACAA
+ // T0: AATGTG--AACAA T0: AATGTG--AACAA
+ // T-2: AA--TGTGAATAA T-2: AA--TGTGAATAA:
+ //
+ // On left: both T0 and T-2 are the same length, as it's
+ // just a deletion that moved. We may end up assigning
+ // reads to an indel allele based on the SNP they have and
+ // not the actual indel.
+ // There *is* a deletion here though, but only 1. How do
+ // we call it once only? Need to replace entire region
+ // with a reassembly.
+ //
+ // On right: T0 and T-2 have same length again, but there
+ // isn't an indel as it's ins+del vs del+ins. They're
+ // also the same length as the REF for this region.
+ // Hence likelihood of this variant existing is tied in
+ // with their equal and high similarity with/to the ref.
+ //
+ // We could do an alignment of tcons[0] and tcons[1] and check
+ // whether their differences are consistent with (ie the
+ // hamming distance is at least ABS(types[t]/2). I don't think
+ // it'll rescue many FPs though.
+
+#ifdef CONS_DEBUG
+ {
+ int j;
+ for (j = 0; j < 2; j++) {
+ int k;
+ fprintf(bcftools_stderr, "Cons%d @ %d %4d/%4d ",
+ j, pos, types[t], left_shift);
+ for (k = 0; k < tcon_len[j]; k++) {
+ if (k == cpos_pos)
+ putc('#', bcftools_stderr);
+ putc("ACGTN"[(uint8_t)tcons[j][k]], bcftools_stderr);
+ }
+ putc('\n', bcftools_stderr);
+ }
+ }
+#endif
+
+ // Scan for base-runs in the insertion.
+ // We use this to avoid over-correction in est_seqQ when the
+ // insertion is not part of the neighbouring homopolymer.
+ int k = tcons[0][cpos_pos], j;
+ for (j = 0; j < types[t]; j++)
+ if (tcons[0][cpos_pos+j] != k)
+ break;
+ if (j && j == types[t])
+ l_run_ins |= "\x1\x2\x4\x8\xf"[k]; // ACGTN
+ if (types[t] < 0)
+ l_run_ins |= 0xff;
+
+ // align each read to consensus(es)
+ for (i = 0; i < n_plp[s]; ++i, ++K) {
+ bam_pileup1_t *p = plp[s] + i;
+
+ // Some basic ref vs alt stats.
+ int imq = p->b->core.qual > 59 ? 59 : p->b->core.qual;
+ imq *= nqual_over_60;
+
+ int sc_len, slen, epos, sc_end;
+
+ // Only need to gather stats on one type, as it's
+ // identical calculation for all the subsequent ones
+ // and we're sharing the same stats array
+ if (t == 0) {
+ // Gather stats for INFO field to aid filtering.
+ // mq and sc_len not very helpful for filtering, but could
+ // help in assigning a better QUAL value.
+ //
+ // Pos is slightly useful.
+ // Base qual can be useful, but need qual prior to BAQ?
+ // May need to cache orig quals in aux tag so we can fetch
+ // them even after mpileup step.
+ get_pos(bca, p, &sc_len, &slen, &epos, &sc_end);
+
+ assert(imq >= 0 && imq < bca->nqual);
+ assert(epos >= 0 && epos < bca->npos);
+ assert(sc_len >= 0 && sc_len < 100);
+ if (p->indel) {
+ bca->ialt_mq[imq]++;
+ bca->ialt_scl[sc_len]++;
+ bca->ialt_pos[epos]++;
+ } else {
+ bca->iref_mq[imq]++;
+ bca->iref_scl[sc_len]++;
+ bca->iref_pos[epos]++;
+ }
+ }
+
+ int qbeg, qpos, qend, tbeg, tend, kk;
+ uint8_t *seq = bam_get_seq(p->b);
+ uint32_t *cigar = bam_get_cigar(p->b);
+ if (p->b->core.flag & BAM_FUNMAP) continue;
+
+ // FIXME: the following loop should be better moved outside;
+ // nonetheless, realignment should be much slower anyway.
+ for (kk = 0; kk < p->b->core.n_cigar; ++kk)
+ if ((cigar[kk]&BAM_CIGAR_MASK) == BAM_CREF_SKIP)
+ break;
+ if (kk < p->b->core.n_cigar)
+ continue;
+
+ // determine the start and end of sequences for alignment
+ int left2 = left, right2 = right;
+ int min_win_size = MAX(-biggest_del, biggest_ins);
+ min_win_size += ABS(left_shift) + ABS(right_shift);
+ {
+ rep_ele *reps, *elt, *tmp;
+ reps = find_STR(tcons[0], tcon_len[0], 0);
+ //int max_str = 0;
+ int tot_str = 0;
+ DL_FOREACH_SAFE(reps, elt, tmp) {
+ // if (max_str < elt->end - elt->start)
+ // max_str = elt->end - elt->start;
+ tot_str += elt->end - elt->start;
+ DL_DELETE(reps, elt);
+ free(elt);
+ }
+
+ // Ideally max_str should be enough, but it's still not
+ // sufficient in longer range some repeats.
+ //min_win_size += max_str;
+ min_win_size += tot_str;
+ }
+ min_win_size += 10;
+
+// TEST 8
+ if (p->b->core.l_qseq > 1000) {
+ // long read data needs less context. It also tends to
+ // have many more candidate indels to investigate so
+ // speed here matters more.
+ if (pos - left >= min_win_size)
+ left2 = MAX(left2, pos - min_win_size);
+ if (right-pos >= min_win_size)
+ right2 = MIN(right2, pos + min_win_size);
+ }
+
+ // Genomic coords for first and last base of query
+ // alignment. This is only used in bcf_cgp_align_score
+ // for computing scores by looking for the proximity
+ // of STRs with the end of the query alignment.
+ int r_start = p->b->core.pos;
+ int r_end = bam_cigar2rlen(p->b->core.n_cigar,
+ bam_get_cigar(p->b));
+ r_end += -1 + r_start;
+
+
+ // Map left2/right2 genomic coordinates to qbeg/qend
+ // query coordinates. The query may not span the
+ // entire left/right region, so this also returns the
+ // equivalent genomic coords for qbeg/qend in tbeg/tend.
+ qbeg = tpos2qpos(&p->b->core, bam_get_cigar(p->b),
+ left2, 0, &tbeg);
+ qpos = tpos2qpos(&p->b->core, bam_get_cigar(p->b), pos,
+ 0, &tend) - qbeg;
+ qend = tpos2qpos(&p->b->core, bam_get_cigar(p->b),
+ right2, 1, &tend);
+
+ int old_tend = tend;
+ int old_tbeg = tbeg;
+
+ // write the query sequence
+ for (l = qbeg; l < qend; ++l)
+ query[l - qbeg] = seq_nt16_int[bam_seqi(seq, l)];
+
+ // tbeg and tend are the genomic locations equivalent
+ // to qbeg and qend on the sequence.
+ // These may being entirely within our left/right
+ // coordinates over which we've computed the
+ // consensus, or overlapping to left/right.
+ //
+ // We know an estimation of band, plus biggest indel,
+ // so we can trim tbeg/tend to a smaller region if we
+ // wish here. This speeds up BAQ scoring.
+ int wband = band + MAX(-biggest_del, biggest_ins)*2 + 20;
+ int tend1 = left + tcon_len[0] - (left2-left);
+ int tend2 = left + tcon_len[1] - (left2-left);
+ tend1 = MIN(tend1, old_tend + wband);
+ tend2 = MIN(tend2, old_tend + wband);
+ tbeg = MAX(left2, old_tbeg - wband);
+
+ // do realignment; this is the bottleneck.
+ //
+ // Note low score = good, high score = bad.
+ if (tend1 > tbeg && tend2 > tbeg) {
+ //fprintf(bcftools_stderr, "Num %d\n", i);
+ if (bcf_cgp_align_score(p, bca, types[t], band,
+ (uint8_t *)tcons[0] + left2-left,
+ (uint8_t *)tcons[1] + left2-left,
+ (uint8_t *)query,
+ r_start, r_end,
+ tbeg, tend1, tend2,
+ left2, left + tcon_len[0],
+ qbeg, qend, pos,qpos, -biggest_del,
+ qavg, bca->del_bias,
+ &score[K*n_types + t],
+ &str_len1, &str_len2) < 0) {
+ goto err;
+ }
+#ifdef ALIGN_DEBUG
+ fprintf(bcftools_stderr, "type %d %x / %x\t%s\n",
+ types[t],
+ score[K*n_types + t] >> 8,
+ score[K*n_types + t] & 0xff,
+ bam_get_qname(p->b));
+#endif
+ } else {
+ // place holder large cost for reads that cover the
+ // region entirely within a deletion (thus tend < tbeg).
+ score[K*n_types + t] = 0xffffff;
+ }
+ }
+ free(tcons);
+ }
+ }
+
+ // compute indelQ
+ if (!(l_run_base & l_run_ins))
+ l_run = 1; // different base type in ins to flanking region.
+ n_alt = bcf_cgp_compute_indelQ(n, n_plp, plp, bca, inscns, l_run, max_ins,
+ ref_type, types, n_types, qavg, score,
+ str_len1, str_len2);
+
+ err:
+ // free
+ free(query);
+ free(score);
+ free(types);
+ free(inscns);
+
+ return n_alt > 0? 0 : -1;
+}
// Finds the smallest index in the seq_pos array holding value equal to pos, or if there is no
// such value, the largest index with value smaller than pos. Starts at initial guess ioff.
// This could use a binary search but the assumption is that the initial guess is indel-size close
-// to the actuall coordinate.
+// to the actual coordinate.
//
// TODO: remove this function and seq_pos from cns creation as it seems unnecessary
static int find_ref_offset(hts_pos_t pos, hts_pos_t *seq_pos, int nseq_pos, int ioff)
// Finds the smallest index in the seq_pos array holding value equal to pos, or if there is no
// such value, the largest index with value smaller than pos. Starts at initial guess ioff.
// This could use a binary search but the assumption is that the initial guess is indel-size close
-// to the actuall coordinate.
+// to the actual coordinate.
//
// TODO: remove this function and seq_pos from cns creation as it seems unnecessary
static int find_ref_offset(hts_pos_t pos, hts_pos_t *seq_pos, int nseq_pos, int ioff)
//
// *_tpos is returned as tpos if query overlaps tpos, but for deletions
// it'll be either the start (is_left) or end (!is_left) ref position.
-static int tpos2qpos(const bam1_core_t *c, const uint32_t *cigar, int32_t tpos, int is_left, int32_t *_tpos)
+int tpos2qpos(const bam1_core_t *c, const uint32_t *cigar, int32_t tpos, int is_left, int32_t *_tpos)
{
// x = pos in ref, y = pos in query seq
int k, x = c->pos, y = 0, last_y = 0;
}
// Identify spft-clip length, position in seq, and clipped seq len
-static inline void get_pos(const bcf_callaux_t *bca, bam_pileup1_t *p,
- int *sc_len_r, int *slen_r, int *epos_r, int *end) {
+void get_pos(const bcf_callaux_t *bca, bam_pileup1_t *p,
+ int *sc_len_r, int *slen_r, int *epos_r, int *end) {
bam1_t *b = p->b;
int sc_len = 0, sc_dist = -1, at_left = 1;
int epos = p->qpos, slen = b->core.l_qseq;
//
// Scans the pileup to identify all the different sizes of indels
// present.
+// types[] returned is sorted by size, from smallest (maybe negative) to largest.
//
// Returns types and fills out n_types_r, max_rd_len_r and ref_type_r,
// or NULL on error.
// Compute the consensus for this sample 's', minus indels which
// get added later.
-static char *bcf_cgp_calc_cons(int n, int *n_plp, bam_pileup1_t **plp,
- int pos, int *types, int n_types,
- int max_ins, int s) {
+char *bcf_cgp_calc_cons(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, int *types, int n_types,
+ int max_ins, int s) {
int i, j, t, k;
int *inscns_aux = (int*)calloc(5 * n_types * max_ins, sizeof(int));
if (!inscns_aux)
//
// *_tpos is returned as tpos if query overlaps tpos, but for deletions
// it'll be either the start (is_left) or end (!is_left) ref position.
-static int tpos2qpos(const bam1_core_t *c, const uint32_t *cigar, int32_t tpos, int is_left, int32_t *_tpos)
+int tpos2qpos(const bam1_core_t *c, const uint32_t *cigar, int32_t tpos, int is_left, int32_t *_tpos)
{
// x = pos in ref, y = pos in query seq
int k, x = c->pos, y = 0, last_y = 0;
}
// Identify spft-clip length, position in seq, and clipped seq len
-static inline void get_pos(const bcf_callaux_t *bca, bam_pileup1_t *p,
- int *sc_len_r, int *slen_r, int *epos_r, int *end) {
+void get_pos(const bcf_callaux_t *bca, bam_pileup1_t *p,
+ int *sc_len_r, int *slen_r, int *epos_r, int *end) {
bam1_t *b = p->b;
int sc_len = 0, sc_dist = -1, at_left = 1;
int epos = p->qpos, slen = b->core.l_qseq;
//
// Scans the pileup to identify all the different sizes of indels
// present.
+// types[] returned is sorted by size, from smallest (maybe negative) to largest.
//
// Returns types and fills out n_types_r, max_rd_len_r and ref_type_r,
// or NULL on error.
// Compute the consensus for this sample 's', minus indels which
// get added later.
-static char *bcf_cgp_calc_cons(int n, int *n_plp, bam_pileup1_t **plp,
- int pos, int *types, int n_types,
- int max_ins, int s) {
+char *bcf_cgp_calc_cons(int n, int *n_plp, bam_pileup1_t **plp,
+ int pos, int *types, int n_types,
+ int max_ins, int s) {
int i, j, t, k;
int *inscns_aux = (int*)calloc(5 * n_types * max_ins, sizeof(int));
if (!inscns_aux)
/* bcftools.h -- utility function declarations.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/hts_defs.h>
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
+#include <htslib/kfunc.h>
#include <math.h>
#define FT_TAB_TEXT 0 // custom tab-delimited text file
void error_errno(const char *format, ...) HTS_NORETURN HTS_FORMAT(HTS_PRINTF_FMT, 1, 2);
// For on the fly index creation with --write-index
-int init_index(htsFile *fh, bcf_hdr_t *hdr, char *fname, char **idx_fname);
+int init_index2(htsFile *fh, bcf_hdr_t *hdr, const char *fname, char **idx_fname, int idx_fmt);
+int init_index(htsFile *fh, bcf_hdr_t *hdr, const char *fname, char **idx_fname);
+
+// Used to set args->write_index in CLI.
+// It will be true if set correctly.
+// Note due to HTS_FMT_CSI being zero we have to use an additional bit.
+int write_index_parse(char *arg);
void bcf_hdr_append_version(bcf_hdr_t *hdr, int argc, char **argv, const char *cmd);
const char *hts_bcf_wmode(int file_type);
int read_AF(bcf_sr_regions_t *tgt, bcf1_t *line, double *alt_freq);
int parse_overlap_option(const char *arg);
+// Default sort order: chr,pos,alleles
+int cmp_bcf_pos(const void *aptr, const void *bptr);
+int cmp_bcf_pos_ref_alt(const void *aptr, const void *bptr);
+
static inline int iupac2bitmask(char iupac)
{
const int A = 1;
return prob>99 ? 99 : prob;
}
+static inline double calc_binom_two_sided(int na, int nb, double aprob)
+{
+ if ( !na && !nb ) return -1;
+ if ( na==nb ) return 1;
+
+ // kfunc.h implements kf_betai, which is the regularized beta function P(X<=k/N;p) = I_{1-p}(N-k,k+1)
+
+ double prob = na > nb ? 2 * kf_betai(na, nb+1, aprob) : 2 * kf_betai(nb, na+1, aprob);
+
+ if ( prob > 1 ) prob = 1; // this can happen, machine precision error, eg. kf_betai(1,0,0.5)
+ return prob;
+}
+static inline double calc_binom_one_sided(int na, int nb, double aprob, int ge)
+{
+ return ge ? kf_betai(na, nb + 1, aprob) : kf_betai(nb, na + 1, 1 - aprob);
+}
+
static const uint64_t bcf_double_missing = 0x7ff0000000000001;
static const uint64_t bcf_double_vector_end = 0x7ff0000000000002;
static inline void bcf_double_set(double *ptr, uint64_t value)
#define bcf_double_is_missing(x) bcf_double_test((x),bcf_double_missing)
#define bcf_double_is_missing_or_vector_end(x) (bcf_double_test((x),bcf_double_missing) || bcf_double_test((x),bcf_double_vector_end))
+static inline int get_unseen_allele(bcf1_t *line)
+{
+ int i;
+ for (i=1; i<line->n_allele; i++)
+ {
+ if ( !strcmp(line->d.allele[i],"<*>") ) return i;
+ if ( !strcmp(line->d.allele[i],"<NON_REF>") ) return i;
+ if ( !strcmp(line->d.allele[i],"<X>") ) return i;
+ }
+ return 0;
+}
+
#endif
#define CALL_VARONLY (1<<1)
#define CALL_CONSTR_TRIO (1<<2)
#define CALL_CONSTR_ALLELES (1<<3)
-//
+#define CALL_KEEP_UNSEEN (1<<4)
#define CALL_FMT_PV4 (1<<5)
#define CALL_FMT_GQ (1<<6)
#define CALL_FMT_GP (1<<7)
void error(const char *format, ...);
/*
- * call() - return -1 value on critical error; -2 to skip the site; or the number of non-reference
- * alleles on success.
+ * call() - return -1 value on critical error; -2 to skip the site; or the number of alleles on success
*/
int mcall(call_t *call, bcf1_t *rec); // multiallic and rare-variant calling model
int ccall(call_t *call, bcf1_t *rec); // the default consensus calling model
/* The MIT License
- Copyright (c) 2014-2023 Genome Research Ltd.
+ Copyright (c) 2014-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
char **argv;
int argc, output_iupac, iupac_GTs, haplotype, allele, isample, napplied;
uint8_t *iupac_bitmask, *iupac_als;
- int miupac_bitmask, miupac_als;
+ int miupac_bitmask, miupac_als, regions_overlap;
char *fname, *ref_fname, *sample, *sample_fname, *output_fname, *mask_fname, *chain_fname, missing_allele, absent_allele;
char mark_del, mark_ins, mark_snv;
smpl_ilist_t *smpl;
args->hdr = args->files->readers[0].header;
args->isample = -1;
if ( !args->sample )
+ {
args->smpl = smpl_ilist_init(args->hdr,NULL,0,SMPL_NONE|SMPL_VERBOSE);
+ if ( !args->smpl->n )
+ {
+ smpl_ilist_destroy(args->smpl);
+ args->smpl = NULL;
+ }
+ }
else if ( args->sample && strcmp("-",args->sample) )
{
args->smpl = smpl_ilist_init(args->hdr,args->sample,0,SMPL_NONE|SMPL_VERBOSE);
{
if ( args->haplotype || args->allele )
{
- if ( args->smpl->n > 1 ) error("Too many samples, only one can be used with -H\n");
+ if ( args->smpl->n > 1 ) error("Too many samples, only one can be used with -H; check the -s,-S options\n");
args->isample = args->smpl->idx[0];
}
else
+ {
args->iupac_GTs = 1;
+ if ( args->smpl->n==1 )
+ fprintf(stderr,"Note: applying IUPAC codes based on FORMAT/GT in sample %s\n",bcf_hdr_int2id(args->hdr,BCF_DT_SAMPLE,args->smpl->idx[0]));
+ else
+ fprintf(stderr,"Note: applying IUPAC codes based on FORMAT/GT in %d samples\n",args->smpl->n);
+ }
}
+ else if ( args->output_iupac )
+ fprintf(stderr,"Note: applying IUPAC codes based on REF,ALT%s\n",bcf_hdr_nsamples(args->hdr)?", ignoring samples":"");
+ else
+ fprintf(stderr,"Note: applying REF,ALT variants%s\n",bcf_hdr_nsamples(args->hdr)?", ignoring samples":"");
int i;
for (i=0; i<args->nmask; i++)
{
if ( ! args->fp_out ) error("Failed to create %s: %s\n", args->output_fname, strerror(errno));
}
else args->fp_out = stdout;
- if ( args->isample<0 && !args->iupac_GTs ) fprintf(stderr,"Note: the --samples option not given, applying all records regardless of the genotype\n");
if ( args->filter_str )
args->filter = filter_init(args->hdr, args->filter_str);
args->rid = -1;
args->prev_base_pos = -1;
args->fa_buf.l = 0;
args->fa_length = 0;
- args->fa_end_pos = to;
- args->fa_ori_pos = from;
- args->fa_src_pos = from;
+ args->fa_end_pos = to; // 0-based
+ args->fa_ori_pos = from; // 0-based
+ args->fa_src_pos = from; // 0-based
args->fa_mod_off = 0;
args->fa_frz_pos = -1;
args->fa_frz_mod = -1;
args->fa_case = -1;
args->vcf_rbuf.n = 0;
+
+ // bcf_sr_set_regions accepts 1-based coordinates
kstring_t str = {0,0,0};
- if ( from==0 ) from = 1;
- if ( to==0 ) to = HTS_POS_MAX;
+ if ( !from ) from = 1;
+ else from++;
+#ifndef MAX_CSI_COOR
+#define MAX_CSI_COOR ((1LL << (14 + 30)) - 1)
+#endif
+ if ( to==0 ) to = MAX_CSI_COOR - 1;
+ else to++;
ksprintf(&str,"%s:%"PRIhts_pos"-%"PRIhts_pos,line,from,to);
- bcf_sr_set_regions(args->files,line,0);
+ bcf_sr_set_opt(args->files,BCF_SR_REGIONS_OVERLAP,args->regions_overlap);
+ bcf_sr_set_regions(args->files,str.s,0);
free(str.s);
if ( tmp_ptr ) *tmp_ptr = tmp;
else if ( (var_type & VCF_OTHER) && !strncasecmp(rec->d.allele[ialt],"<INS",4) ) trim_beg = 1;
// Overlapping variant?
+ if ( ialt==0 && rec->pos <= args->fa_frz_pos && rec->pos + rec->rlen - 1 > args->fa_frz_pos )
+ {
+ // Applying the reference allele which overlaps a previous deletion. If we are here, it
+ // means it goes beyond the freezed position, hence the record can be trimmed and moved
+ // forward
+ int ntrim = args->fa_frz_pos - rec->pos + 1;
+ int nref = strlen(rec->d.allele[0]);
+ assert( ntrim < nref );
+ rec->pos += ntrim;
+ rec->rlen -= ntrim;
+ memmove(rec->d.allele[0],rec->d.allele[0]+ntrim,nref-ntrim);
+ rec->d.allele[0][nref-ntrim] = 0;
+ }
if ( rec->pos <= args->fa_frz_pos )
{
// Can be still OK iff this is an insertion (and which does not follow another insertion, see #888).
fprintf(stderr,"The site %s:%"PRId64" overlaps with another variant, skipping...\n", bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1);
return;
}
-
}
+ char *ref_allele = rec->d.allele[0];
char *alt_allele = rec->d.allele[ialt];
int rmme_alt = 0;
int len_diff = 0, alen = 0;
- int idx = rec->pos - args->fa_ori_pos + args->fa_mod_off;
+ int idx = rec->pos - args->fa_ori_pos + args->fa_mod_off; // position of the variant within the modified fasta sequence
if ( idx<0 )
{
- fprintf(stderr,"Warning: ignoring overlapping variant starting at %s:%"PRId64"\n", bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1);
- return;
+ if ( alt_allele[0]=='<' ) // symbolic allele
+ {
+ rec->pos -= idx + 1;
+ rec->rlen += idx + 1;
+ idx = -1;
+ }
+ else if ( strlen(ref_allele) < -idx ) // the ref allele is shorter but overlaps the fa sequence? This should never happen
+ {
+ assert(0);
+ fprintf(stderr,"Warning: ignoring overlapping variant starting at %s:%"PRId64"\n", bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1);
+ return;
+ }
+ else if ( strlen(alt_allele) > -idx ) // the ref allele overlaps the fa and so does the alt allele
+ {
+ rec->pos -= idx;
+ rec->rlen += idx;
+ ref_allele -= idx;
+ alt_allele -= idx;
+ idx = 0;
+ }
+ else // the ref allele overlaps the fa but alt allele does not: trim to leave one base before
+ {
+ rec->pos -= idx + 1;
+ rec->rlen += idx + 1;
+ ref_allele -= idx + 1;
+ alt_allele += strlen(alt_allele) - 1;
+ idx = -1;
+ }
}
if ( rec->rlen > args->fa_buf.l - idx )
{
}
}
}
- if ( idx>=args->fa_buf.l )
- error("FIXME: %s:%"PRId64" .. idx=%d, ori_pos=%d, len=%"PRIu64", off=%d\n",bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1,idx,args->fa_ori_pos,(uint64_t)args->fa_buf.l,args->fa_mod_off);
+
+ // the variant is beyond the available fasta sequence
+ if ( idx>0 && idx>=args->fa_buf.l ) return;
// sanity check the reference base
if ( alt_allele[0]=='<' )
if ( !strcasecmp(alt_allele,"<DEL>") )
{
static int multibase_ref_del_warned = 0;
- if ( rec->d.allele[0][1]!=0 && !multibase_ref_del_warned )
+ if ( ref_allele[1]!=0 && !multibase_ref_del_warned )
{
fprintf(stderr,
"Warning: one REF base is expected with <DEL>, assuming the actual deletion starts at POS+1 at %s:%"PRId64".\n"
}
if ( args->mark_del ) // insert dashes instead of delete sequence
{
- alt_allele = mark_del(rec->d.allele[0], rec->rlen, NULL, args->mark_del);
+ alt_allele = mark_del(ref_allele, rec->rlen, NULL, args->mark_del);
alen = rec->rlen;
len_diff = 0;
rmme_alt = 1;
else
{
len_diff = 1-rec->rlen;
- alt_allele = rec->d.allele[0]; // according to VCF spec, the first REF base must precede the event
+ alt_allele = ref_allele; // according to VCF spec, the first REF base must precede the event
alen = 1;
}
}
return;
}
}
- else if ( strncasecmp(rec->d.allele[0],args->fa_buf.s+idx,rec->rlen) )
+ else if ( idx>=0 && strncasecmp(ref_allele,args->fa_buf.s+idx,rec->rlen) )
{
// This is hacky, handle a special case: if SNP or an insert follows a deletion (AAC>A, C>CAA),
// the reference base in fa_buf is lost and the check fails. We do not keep a buffer
// one base overlap
int fail = 1;
- if ( args->prev_base_pos==rec->pos && toupper(rec->d.allele[0][0])==toupper(args->prev_base) )
+ if ( args->prev_base_pos==rec->pos && toupper(ref_allele[0])==toupper(args->prev_base) )
{
if ( rec->rlen==1 ) fail = 0;
- else if ( !strncasecmp(rec->d.allele[0]+1,args->fa_buf.s+idx+1,rec->rlen-1) ) fail = 0;
+ else if ( !strncasecmp(ref_allele+1,args->fa_buf.s+idx+1,rec->rlen-1) ) fail = 0;
}
if ( fail )
" REF .vcf: [%s]\n"
" ALT .vcf: [%s]\n"
" REF .fa : [%s]%c%s\n",
- bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1, rec->d.allele[0], alt_allele, args->fa_buf.s+idx,
+ bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1, ref_allele, alt_allele, args->fa_buf.s+idx,
tmp?tmp:' ',tmp?args->fa_buf.s+idx+rec->rlen+1:""
);
}
if ( args->mark_del && len_diff<0 )
{
- alt_allele = mark_del(rec->d.allele[0], rec->rlen, alt_allele, args->mark_del);
+ alt_allele = mark_del(ref_allele, rec->rlen, alt_allele, args->mark_del);
alen = rec->rlen;
len_diff = 0;
rmme_alt = 1;
if ( args->mark_del && len_diff<0 )
{
- alt_allele = mark_del(rec->d.allele[0], rec->rlen, alt_allele, args->mark_del);
+ alt_allele = mark_del(ref_allele, rec->rlen, alt_allele, args->mark_del);
alen = rec->rlen;
len_diff = 0;
rmme_alt = 1;
}
}
- args->fa_case = toupper(args->fa_buf.s[idx])==args->fa_buf.s[idx] ? TO_UPPER : TO_LOWER;
+ int safe_idx = idx<0 ? 0 : idx; // idx can be negative in case of overlapping deletion
+ args->fa_case = toupper(args->fa_buf.s[safe_idx])==args->fa_buf.s[safe_idx] ? TO_UPPER : TO_LOWER;
if ( args->fa_case==TO_UPPER )
for (i=0; i<alen; i++) alt_allele[i] = toupper(alt_allele[i]);
else
for (i=0; i<alen; i++) alt_allele[i] = tolower(alt_allele[i]);
if ( args->mark_ins && len_diff>0 )
- mark_ins(rec->d.allele[0], alt_allele, args->mark_ins);
+ mark_ins(ref_allele, alt_allele, args->mark_ins);
if ( args->mark_snv )
- mark_snv(rec->d.allele[0], alt_allele, args->mark_snv);
+ mark_snv(ref_allele, alt_allele, args->mark_snv);
if ( len_diff <= 0 )
{
ks_resize(&args->fa_buf, args->fa_buf.l + len_diff);
memmove(args->fa_buf.s + idx + rec->rlen + len_diff, args->fa_buf.s + idx + rec->rlen, args->fa_buf.l - idx - rec->rlen);
- // This can get tricky, make sure the bases unchanged by the insertion do not overwrite preceeding variants.
+ // This can get tricky, make sure the bases unchanged by the insertion do not overwrite preceding variants.
// For example, here we want to get TAA:
// POS REF ALT
// 1 C T
// 1 C CAA
int ibeg = 0;
- while ( ibeg<alen && rec->d.allele[0][ibeg]==alt_allele[ibeg] && rec->pos + ibeg <= args->prev_base_pos ) ibeg++;
+ while ( ibeg<alen && ref_allele[ibeg]==alt_allele[ibeg] && rec->pos + ibeg <= args->prev_base_pos ) ibeg++;
for (i=ibeg; i<alen; i++)
args->fa_buf.s[idx+i] = alt_allele[i];
if (args->chain && len_diff != 0)
{
// If first nucleotide of both REF and ALT are the same... (indels typically include the nucleotide before the variant)
- if ( strncasecmp(rec->d.allele[0],alt_allele,1) == 0)
+ if ( strncasecmp(ref_allele,alt_allele,1) == 0)
{
// ...extend the block by 1 bp: start is 1 bp further and alleles are 1 bp shorter
push_chain_gap(args->chain, rec->pos + 1, rec->rlen - 1, rec->pos + 1 + args->fa_mod_off, alen - 1);
fprintf(stderr, " -M, --missing CHAR Output CHAR instead of skipping a missing genotype \"./.\"\n");
fprintf(stderr, " -o, --output FILE Write output to a file [standard output]\n");
fprintf(stderr, " -p, --prefix STRING Prefix to add to output sequence names\n");
+ fprintf(stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
fprintf(stderr, " -s, --samples LIST Comma-separated list of samples to include, \"-\" to ignore samples and use REF,ALT\n");
fprintf(stderr, " -S, --samples-file FILE File of samples to include\n");
fprintf(stderr, "Examples:\n");
{
args_t *args = (args_t*) calloc(1,sizeof(args_t));
args->argc = argc; args->argv = argv;
+ args->regions_overlap = 1;
static struct option loptions[] =
{
{"absent",1,0,'a'},
{"chain",1,0,'c'},
{"prefix",required_argument,0,'p'},
+ {"regions-overlap",required_argument,0,5},
{0,0,0,0}
};
int c;
else if ( !optarg[1] && optarg[0]>32 && optarg[0]<127 ) args->mark_snv = optarg[0];
else error("The argument is not recognised: --mark-snv %s\n",optarg);
break;
+ case 5 :
+ args->regions_overlap = parse_overlap_option(optarg);
+ if ( args->regions_overlap < 0 ) error("Could not parse: --regions-overlap %s\n",optarg);
+ break;
case 'p': args->chr_prefix = optarg; break;
case 's': args->sample = optarg; break;
case 'S': args->sample_fname = optarg; break;
/* The MIT License
- Copyright (c) 2014-2023 Genome Research Ltd.
+ Copyright (c) 2014-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
char **argv;
int argc, output_iupac, iupac_GTs, haplotype, allele, isample, napplied;
uint8_t *iupac_bitmask, *iupac_als;
- int miupac_bitmask, miupac_als;
+ int miupac_bitmask, miupac_als, regions_overlap;
char *fname, *ref_fname, *sample, *sample_fname, *output_fname, *mask_fname, *chain_fname, missing_allele, absent_allele;
char mark_del, mark_ins, mark_snv;
smpl_ilist_t *smpl;
args->hdr = args->files->readers[0].header;
args->isample = -1;
if ( !args->sample )
+ {
args->smpl = smpl_ilist_init(args->hdr,NULL,0,SMPL_NONE|SMPL_VERBOSE);
+ if ( !args->smpl->n )
+ {
+ smpl_ilist_destroy(args->smpl);
+ args->smpl = NULL;
+ }
+ }
else if ( args->sample && strcmp("-",args->sample) )
{
args->smpl = smpl_ilist_init(args->hdr,args->sample,0,SMPL_NONE|SMPL_VERBOSE);
{
if ( args->haplotype || args->allele )
{
- if ( args->smpl->n > 1 ) error("Too many samples, only one can be used with -H\n");
+ if ( args->smpl->n > 1 ) error("Too many samples, only one can be used with -H; check the -s,-S options\n");
args->isample = args->smpl->idx[0];
}
else
+ {
args->iupac_GTs = 1;
+ if ( args->smpl->n==1 )
+ fprintf(bcftools_stderr,"Note: applying IUPAC codes based on FORMAT/GT in sample %s\n",bcf_hdr_int2id(args->hdr,BCF_DT_SAMPLE,args->smpl->idx[0]));
+ else
+ fprintf(bcftools_stderr,"Note: applying IUPAC codes based on FORMAT/GT in %d samples\n",args->smpl->n);
+ }
}
+ else if ( args->output_iupac )
+ fprintf(bcftools_stderr,"Note: applying IUPAC codes based on REF,ALT%s\n",bcf_hdr_nsamples(args->hdr)?", ignoring samples":"");
+ else
+ fprintf(bcftools_stderr,"Note: applying REF,ALT variants%s\n",bcf_hdr_nsamples(args->hdr)?", ignoring samples":"");
int i;
for (i=0; i<args->nmask; i++)
{
if ( ! args->fp_out ) error("Failed to create %s: %s\n", args->output_fname, strerror(errno));
}
else args->fp_out = bcftools_stdout;
- if ( args->isample<0 && !args->iupac_GTs ) fprintf(bcftools_stderr,"Note: the --samples option not given, applying all records regardless of the genotype\n");
if ( args->filter_str )
args->filter = filter_init(args->hdr, args->filter_str);
args->rid = -1;
args->prev_base_pos = -1;
args->fa_buf.l = 0;
args->fa_length = 0;
- args->fa_end_pos = to;
- args->fa_ori_pos = from;
- args->fa_src_pos = from;
+ args->fa_end_pos = to; // 0-based
+ args->fa_ori_pos = from; // 0-based
+ args->fa_src_pos = from; // 0-based
args->fa_mod_off = 0;
args->fa_frz_pos = -1;
args->fa_frz_mod = -1;
args->fa_case = -1;
args->vcf_rbuf.n = 0;
+
+ // bcf_sr_set_regions accepts 1-based coordinates
kstring_t str = {0,0,0};
- if ( from==0 ) from = 1;
- if ( to==0 ) to = HTS_POS_MAX;
+ if ( !from ) from = 1;
+ else from++;
+#ifndef MAX_CSI_COOR
+#define MAX_CSI_COOR ((1LL << (14 + 30)) - 1)
+#endif
+ if ( to==0 ) to = MAX_CSI_COOR - 1;
+ else to++;
ksprintf(&str,"%s:%"PRIhts_pos"-%"PRIhts_pos,line,from,to);
- bcf_sr_set_regions(args->files,line,0);
+ bcf_sr_set_opt(args->files,BCF_SR_REGIONS_OVERLAP,args->regions_overlap);
+ bcf_sr_set_regions(args->files,str.s,0);
free(str.s);
if ( tmp_ptr ) *tmp_ptr = tmp;
else if ( (var_type & VCF_OTHER) && !strncasecmp(rec->d.allele[ialt],"<INS",4) ) trim_beg = 1;
// Overlapping variant?
+ if ( ialt==0 && rec->pos <= args->fa_frz_pos && rec->pos + rec->rlen - 1 > args->fa_frz_pos )
+ {
+ // Applying the reference allele which overlaps a previous deletion. If we are here, it
+ // means it goes beyond the freezed position, hence the record can be trimmed and moved
+ // forward
+ int ntrim = args->fa_frz_pos - rec->pos + 1;
+ int nref = strlen(rec->d.allele[0]);
+ assert( ntrim < nref );
+ rec->pos += ntrim;
+ rec->rlen -= ntrim;
+ memmove(rec->d.allele[0],rec->d.allele[0]+ntrim,nref-ntrim);
+ rec->d.allele[0][nref-ntrim] = 0;
+ }
if ( rec->pos <= args->fa_frz_pos )
{
// Can be still OK iff this is an insertion (and which does not follow another insertion, see #888).
fprintf(bcftools_stderr,"The site %s:%"PRId64" overlaps with another variant, skipping...\n", bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1);
return;
}
-
}
+ char *ref_allele = rec->d.allele[0];
char *alt_allele = rec->d.allele[ialt];
int rmme_alt = 0;
int len_diff = 0, alen = 0;
- int idx = rec->pos - args->fa_ori_pos + args->fa_mod_off;
+ int idx = rec->pos - args->fa_ori_pos + args->fa_mod_off; // position of the variant within the modified fasta sequence
if ( idx<0 )
{
- fprintf(bcftools_stderr,"Warning: ignoring overlapping variant starting at %s:%"PRId64"\n", bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1);
- return;
+ if ( alt_allele[0]=='<' ) // symbolic allele
+ {
+ rec->pos -= idx + 1;
+ rec->rlen += idx + 1;
+ idx = -1;
+ }
+ else if ( strlen(ref_allele) < -idx ) // the ref allele is shorter but overlaps the fa sequence? This should never happen
+ {
+ assert(0);
+ fprintf(bcftools_stderr,"Warning: ignoring overlapping variant starting at %s:%"PRId64"\n", bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1);
+ return;
+ }
+ else if ( strlen(alt_allele) > -idx ) // the ref allele overlaps the fa and so does the alt allele
+ {
+ rec->pos -= idx;
+ rec->rlen += idx;
+ ref_allele -= idx;
+ alt_allele -= idx;
+ idx = 0;
+ }
+ else // the ref allele overlaps the fa but alt allele does not: trim to leave one base before
+ {
+ rec->pos -= idx + 1;
+ rec->rlen += idx + 1;
+ ref_allele -= idx + 1;
+ alt_allele += strlen(alt_allele) - 1;
+ idx = -1;
+ }
}
if ( rec->rlen > args->fa_buf.l - idx )
{
}
}
}
- if ( idx>=args->fa_buf.l )
- error("FIXME: %s:%"PRId64" .. idx=%d, ori_pos=%d, len=%"PRIu64", off=%d\n",bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1,idx,args->fa_ori_pos,(uint64_t)args->fa_buf.l,args->fa_mod_off);
+
+ // the variant is beyond the available fasta sequence
+ if ( idx>0 && idx>=args->fa_buf.l ) return;
// sanity check the reference base
if ( alt_allele[0]=='<' )
if ( !strcasecmp(alt_allele,"<DEL>") )
{
static int multibase_ref_del_warned = 0;
- if ( rec->d.allele[0][1]!=0 && !multibase_ref_del_warned )
+ if ( ref_allele[1]!=0 && !multibase_ref_del_warned )
{
fprintf(bcftools_stderr,
"Warning: one REF base is expected with <DEL>, assuming the actual deletion starts at POS+1 at %s:%"PRId64".\n"
}
if ( args->mark_del ) // insert dashes instead of delete sequence
{
- alt_allele = mark_del(rec->d.allele[0], rec->rlen, NULL, args->mark_del);
+ alt_allele = mark_del(ref_allele, rec->rlen, NULL, args->mark_del);
alen = rec->rlen;
len_diff = 0;
rmme_alt = 1;
else
{
len_diff = 1-rec->rlen;
- alt_allele = rec->d.allele[0]; // according to VCF spec, the first REF base must precede the event
+ alt_allele = ref_allele; // according to VCF spec, the first REF base must precede the event
alen = 1;
}
}
return;
}
}
- else if ( strncasecmp(rec->d.allele[0],args->fa_buf.s+idx,rec->rlen) )
+ else if ( idx>=0 && strncasecmp(ref_allele,args->fa_buf.s+idx,rec->rlen) )
{
// This is hacky, handle a special case: if SNP or an insert follows a deletion (AAC>A, C>CAA),
// the reference base in fa_buf is lost and the check fails. We do not keep a buffer
// one base overlap
int fail = 1;
- if ( args->prev_base_pos==rec->pos && toupper(rec->d.allele[0][0])==toupper(args->prev_base) )
+ if ( args->prev_base_pos==rec->pos && toupper(ref_allele[0])==toupper(args->prev_base) )
{
if ( rec->rlen==1 ) fail = 0;
- else if ( !strncasecmp(rec->d.allele[0]+1,args->fa_buf.s+idx+1,rec->rlen-1) ) fail = 0;
+ else if ( !strncasecmp(ref_allele+1,args->fa_buf.s+idx+1,rec->rlen-1) ) fail = 0;
}
if ( fail )
" REF .vcf: [%s]\n"
" ALT .vcf: [%s]\n"
" REF .fa : [%s]%c%s\n",
- bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1, rec->d.allele[0], alt_allele, args->fa_buf.s+idx,
+ bcf_seqname(args->hdr,rec),(int64_t) rec->pos+1, ref_allele, alt_allele, args->fa_buf.s+idx,
tmp?tmp:' ',tmp?args->fa_buf.s+idx+rec->rlen+1:""
);
}
if ( args->mark_del && len_diff<0 )
{
- alt_allele = mark_del(rec->d.allele[0], rec->rlen, alt_allele, args->mark_del);
+ alt_allele = mark_del(ref_allele, rec->rlen, alt_allele, args->mark_del);
alen = rec->rlen;
len_diff = 0;
rmme_alt = 1;
if ( args->mark_del && len_diff<0 )
{
- alt_allele = mark_del(rec->d.allele[0], rec->rlen, alt_allele, args->mark_del);
+ alt_allele = mark_del(ref_allele, rec->rlen, alt_allele, args->mark_del);
alen = rec->rlen;
len_diff = 0;
rmme_alt = 1;
}
}
- args->fa_case = toupper(args->fa_buf.s[idx])==args->fa_buf.s[idx] ? TO_UPPER : TO_LOWER;
+ int safe_idx = idx<0 ? 0 : idx; // idx can be negative in case of overlapping deletion
+ args->fa_case = toupper(args->fa_buf.s[safe_idx])==args->fa_buf.s[safe_idx] ? TO_UPPER : TO_LOWER;
if ( args->fa_case==TO_UPPER )
for (i=0; i<alen; i++) alt_allele[i] = toupper(alt_allele[i]);
else
for (i=0; i<alen; i++) alt_allele[i] = tolower(alt_allele[i]);
if ( args->mark_ins && len_diff>0 )
- mark_ins(rec->d.allele[0], alt_allele, args->mark_ins);
+ mark_ins(ref_allele, alt_allele, args->mark_ins);
if ( args->mark_snv )
- mark_snv(rec->d.allele[0], alt_allele, args->mark_snv);
+ mark_snv(ref_allele, alt_allele, args->mark_snv);
if ( len_diff <= 0 )
{
ks_resize(&args->fa_buf, args->fa_buf.l + len_diff);
memmove(args->fa_buf.s + idx + rec->rlen + len_diff, args->fa_buf.s + idx + rec->rlen, args->fa_buf.l - idx - rec->rlen);
- // This can get tricky, make sure the bases unchanged by the insertion do not overwrite preceeding variants.
+ // This can get tricky, make sure the bases unchanged by the insertion do not overwrite preceding variants.
// For example, here we want to get TAA:
// POS REF ALT
// 1 C T
// 1 C CAA
int ibeg = 0;
- while ( ibeg<alen && rec->d.allele[0][ibeg]==alt_allele[ibeg] && rec->pos + ibeg <= args->prev_base_pos ) ibeg++;
+ while ( ibeg<alen && ref_allele[ibeg]==alt_allele[ibeg] && rec->pos + ibeg <= args->prev_base_pos ) ibeg++;
for (i=ibeg; i<alen; i++)
args->fa_buf.s[idx+i] = alt_allele[i];
if (args->chain && len_diff != 0)
{
// If first nucleotide of both REF and ALT are the same... (indels typically include the nucleotide before the variant)
- if ( strncasecmp(rec->d.allele[0],alt_allele,1) == 0)
+ if ( strncasecmp(ref_allele,alt_allele,1) == 0)
{
// ...extend the block by 1 bp: start is 1 bp further and alleles are 1 bp shorter
push_chain_gap(args->chain, rec->pos + 1, rec->rlen - 1, rec->pos + 1 + args->fa_mod_off, alen - 1);
fprintf(bcftools_stderr, " -M, --missing CHAR Output CHAR instead of skipping a missing genotype \"./.\"\n");
fprintf(bcftools_stderr, " -o, --output FILE Write output to a file [standard output]\n");
fprintf(bcftools_stderr, " -p, --prefix STRING Prefix to add to output sequence names\n");
+ fprintf(bcftools_stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
fprintf(bcftools_stderr, " -s, --samples LIST Comma-separated list of samples to include, \"-\" to ignore samples and use REF,ALT\n");
fprintf(bcftools_stderr, " -S, --samples-file FILE File of samples to include\n");
fprintf(bcftools_stderr, "Examples:\n");
{
args_t *args = (args_t*) calloc(1,sizeof(args_t));
args->argc = argc; args->argv = argv;
+ args->regions_overlap = 1;
static struct option loptions[] =
{
{"absent",1,0,'a'},
{"chain",1,0,'c'},
{"prefix",required_argument,0,'p'},
+ {"regions-overlap",required_argument,0,5},
{0,0,0,0}
};
int c;
else if ( !optarg[1] && optarg[0]>32 && optarg[0]<127 ) args->mark_snv = optarg[0];
else error("The argument is not recognised: --mark-snv %s\n",optarg);
break;
+ case 5 :
+ args->regions_overlap = parse_overlap_option(optarg);
+ if ( args->regions_overlap < 0 ) error("Could not parse: --regions-overlap %s\n",optarg);
+ break;
case 'p': args->chr_prefix = optarg; break;
case 's': args->sample = optarg; break;
case 'S': args->sample_fname = optarg; break;
/* convert.c -- functions for converting between VCF/BCF and related formats.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <errno.h>
#include <sys/stat.h>
#include <sys/types.h>
+#include <stdint.h>
#define __STDC_FORMAT_MACROS
#include <inttypes.h>
#include <math.h>
#include <htslib/vcfutils.h>
#include <htslib/kfunc.h>
#include <htslib/khash_str2int.h>
+#include <htslib/hts_endian.h>
#include "bcftools.h"
#include "variantkey.h"
#include "convert.h"
char *undef_info_tag;
void *used_tags_hash;
char **used_tags_list;
+ char *print_filtered;
int nused_tags;
int allow_undef_tags;
int force_newline;
+ int header_samples;
+ int no_hdr_indices;
uint8_t **subset_samples;
};
}
else kputc('.', str);
}
-static inline int32_t bcf_array_ivalue(void *bcf_array, int type, int idx)
+static inline int32_t bcf_array_ivalue(uint8_t *bcf_array, int type, int idx)
{
if ( type==BCF_BT_INT8 )
{
- int8_t val = ((int8_t*)bcf_array)[idx];
+ int8_t val = le_to_i8(&bcf_array[idx * sizeof(val)]);
if ( val==bcf_int8_missing ) return bcf_int32_missing;
if ( val==bcf_int8_vector_end ) return bcf_int32_vector_end;
return val;
}
if ( type==BCF_BT_INT16 )
{
- int16_t val = ((int16_t*)bcf_array)[idx];
+ int16_t val = le_to_i16(&bcf_array[idx * sizeof(val)]);
if ( val==bcf_int16_missing ) return bcf_int32_missing;
if ( val==bcf_int16_vector_end ) return bcf_int32_vector_end;
return val;
}
- return ((int32_t*)bcf_array)[idx];
+ return le_to_i32(&bcf_array[idx * sizeof(int32_t)]);
}
static inline void _copy_field(char *src, uint32_t len, int idx, kstring_t *str)
{
kputc('.', str);
return;
}
- #define BRANCH(type_t, is_missing, is_vector_end, kprint) { \
- type_t val = ((type_t *) info->vptr)[fmt->subscript]; \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, kprint) { \
+ type_t val = convert(&info->vptr[fmt->subscript * sizeof(type_t)]); \
if ( is_missing || is_vector_end ) kputc('.',str); \
else kprint; \
}
switch (info->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, val==bcf_int8_missing, val==bcf_int8_vector_end, kputw(val, str)); break;
- case BCF_BT_INT16: BRANCH(int16_t, val==bcf_int16_missing, val==bcf_int16_vector_end, kputw(val, str)); break;
- case BCF_BT_INT32: BRANCH(int32_t, val==bcf_int32_missing, val==bcf_int32_vector_end, kputw(val, str)); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), kputd(val, str)); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, kputw(val, str)); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, kputw(val, str)); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, kputw(val, str)); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), kputd(val, str)); break;
case BCF_BT_CHAR: _copy_field((char*)info->vptr, info->vptr_len, fmt->subscript, str); break;
default: fprintf(stderr,"todo: type %d\n", info->type); exit(1); break;
}
}
if ( fmt->fmt->type == BCF_BT_FLOAT )
{
- float *ptr = (float*)(fmt->fmt->p + isample*fmt->fmt->size);
- if ( bcf_float_is_missing(ptr[fmt->subscript]) || bcf_float_is_vector_end(ptr[fmt->subscript]) )
+ uint8_t *ptr = fmt->fmt->p + isample*fmt->fmt->size;
+ float val = le_to_float(&ptr[fmt->subscript * sizeof(float)]);
+ if ( bcf_float_is_missing(val) || bcf_float_is_vector_end(val) )
kputc('.', str);
else
- kputd(ptr[fmt->subscript], str);
+ kputd(val, str);
}
else if ( fmt->fmt->type != BCF_BT_CHAR )
{
int mask = fmt->subscript==0 ? 3 : 1; // merge both haplotypes if subscript==0
- #define BRANCH(type_t, nbits) { \
- type_t *x = (type_t*)(fmt->fmt->p + isample*fmt->fmt->size); \
+ #define BRANCH(type_t, convert, nbits) { \
+ uint8_t *x = fmt->fmt->p + isample*fmt->fmt->size; \
int i,j; \
if ( fmt->subscript<=0 || fmt->subscript==1 ) \
{ \
for (j=0; j < fmt->fmt->n; j++) \
{ \
- type_t val = x[j]; \
+ type_t val = convert(&x[j * sizeof(type_t)]); \
if ( !val ) continue; \
for (i=0; i<nbits; i+=2) \
if ( val & (mask<<i) ) { kputs(csq->str[(j*30+i)/2], &csq->hap1); kputc_(',', &csq->hap1); } \
{ \
for (j=0; j < fmt->fmt->n; j++) \
{ \
- type_t val = x[j]; \
+ type_t val = convert(&x[j * sizeof(type_t)]); \
if ( !val ) continue; \
for (i=1; i<nbits; i+=2) \
if ( val & (1<<i) ) { kputs(csq->str[(j*30+i)/2], &csq->hap2); kputc_(',', &csq->hap2); } \
}
switch (fmt->fmt->type)
{
- case BCF_BT_INT8: BRANCH(uint8_t, 8); break;
- case BCF_BT_INT16: BRANCH(uint16_t,16); break;
- case BCF_BT_INT32: BRANCH(uint32_t,30); break; // 2 bytes unused to account for the reserved BCF values
+ case BCF_BT_INT8: BRANCH(uint8_t, le_to_u8, 8); break;
+ case BCF_BT_INT16: BRANCH(uint16_t, le_to_u16, 16); break;
+ case BCF_BT_INT32: BRANCH(uint32_t, le_to_u32, 30); break; // 2 bits unused to account for the reserved BCF values
default: error("Unexpected type: %d\n", fmt->fmt->type); exit(1); break;
}
#undef BRANCH
int al = bcf_gt_allele(gt[i]);
if ( al > line->n_allele || al >= fmt->fmt->n ) goto invalid;
- #define BRANCH(type_t, missing, vector_end) { \
- type_t val = ((type_t *) fmt->fmt->p)[al + isample*fmt->fmt->n]; \
+ #define BRANCH(type_t, convert, missing, vector_end) { \
+ type_t val = convert(&fmt->fmt->p[(al + isample*fmt->fmt->n)*sizeof(type_t)]); \
if ( val==missing || val==vector_end ) goto invalid; \
else n[i] = val; \
}
switch (fmt->fmt->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_missing, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_missing, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_missing, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_missing, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_missing, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_missing, bcf_int32_vector_end); break;
default: goto invalid; break;
}
#undef BRANCH
if ( n[0]==n[1] ) kputc(n[0]==0 ? '.':'0', str);
else
{
- double pval = n[0] < n[1] ? kf_betai(n[1], n[0] + 1, 0.5) : kf_betai(n[0], n[1] + 1, 0.5);
- pval *= 2;
- if ( pval>=1 ) pval = 0; // this can happen, machine precision error, eg. kf_betai(1,0,0.5)
- else
- pval = -4.34294481903*log(pval);
+ double pval = calc_binom_two_sided(n[0],n[1],0.5);
+
+ // convrt to phred
+ if ( pval>=1 ) pval = 0;
+ else pval = -4.34294481903*log(pval);
kputd(pval, str);
}
return;
free(convert->used_tags_list);
}
khash_str2int_destroy(convert->used_tags_hash);
+ free(convert->print_filtered);
free(convert->fmt);
free(convert->undef_info_tag);
free(convert->dat);
int convert_header(convert_t *convert, kstring_t *str)
{
int i, icol = 0, l_ori = str->l;
+ bcf_hdr_t *hdr = convert->header;
- // Supress the header output if LINE is present
+ // Suppress the header output if LINE is present
for (i=0; i<convert->nfmt; i++)
if ( convert->fmt[i].type == T_LINE ) break;
if ( i!=convert->nfmt )
while ( convert->fmt[j].is_gt_field ) j++;
for (js=0; js<convert->nsamples; js++)
{
+ int ks = convert->samples[js];
for (k=i; k<j; k++)
{
if ( convert->fmt[k].type == T_SEP )
}
}
}
+ else if ( convert->header_samples )
+ {
+ icol++;
+ if ( !convert->no_hdr_indices ) ksprintf(str,"[%d]",icol);
+ ksprintf(str,"%s:%s", hdr->samples[ks], convert->fmt[k].key);
+ }
else
- ksprintf(str, "[%d]%s", ++icol, convert->fmt[k].key);
+ {
+ icol++;
+ if ( !convert->no_hdr_indices ) ksprintf(str,"[%d]",icol);
+ ksprintf(str,"%s", convert->fmt[k].key);
+ }
+ }
+ if ( has_fmt_newline )
+ {
+ if ( !convert->header_samples ) break;
+
+ // this is unfortunate: the formatting expression breaks the per-sample output into separate lines,
+ // therefore including a sample name in the header makes no sense anymore
+ convert->header_samples = 0;
+ str->l = l_ori;
+ return convert_header(convert, str);
}
- if ( has_fmt_newline ) break;
}
i = j-1;
continue;
if ( convert->fmt[i].key ) kputs(convert->fmt[i].key, str);
continue;
}
- ksprintf(str, "[%d]%s", ++icol, convert->fmt[i].key);
+ icol++;
+ if ( !convert->no_hdr_indices ) ksprintf(str,"[%d]",icol);
+ ksprintf(str,"%s", convert->fmt[i].key);
}
if ( has_fmt_newline ) kputc('\n',str);
return str->l - l_ori;
{
// Skip samples when filtering was requested
int ks = convert->samples[js];
- if ( convert->subset_samples && *convert->subset_samples && !(*convert->subset_samples)[ks] ) continue;
+ if ( convert->subset_samples && *convert->subset_samples && !(*convert->subset_samples)[ks] )
+ {
+ if ( !convert->print_filtered ) continue;
+
+ for (k=i; k<j; k++)
+ if ( convert->fmt[k].type==T_SEP )
+ convert->fmt[k].handler(convert, line, &convert->fmt[k], ks, str);
+ else
+ kputs(convert->print_filtered, str);
+ continue;
+ }
// Here comes a hack designed for TBCSQ. When running on large files,
// such as 1000GP, there are too many empty fields in the output and
}
if ( has_newline ) return;
- // A newline is not present, force it. But where to add it?
- // Consider
- // -f'%CHROM[ %SAMPLE]\n'
- // vs
- // -f'[%CHROM %SAMPLE\n]'
- for (i=0; i<convert->nfmt; i++)
- if ( !convert->fmt[i].is_gt_field && convert->fmt[i].key ) break;
-
- if ( i < convert->nfmt )
- register_tag(convert, "\n", 0, T_SEP); // the first case
- else
- {
- // the second case
- i = convert->nfmt - 1;
- if ( !convert->fmt[i].key )
- {
- convert->fmt[i].key = strdup("\n");
- convert->fmt[i].is_gt_field = 1;
- register_tag(convert, NULL, 0, T_SEP);
- }
- else
- register_tag(convert, "\n", 1, T_SEP);
- }
+ // A newline is not present, force it. But where to add it? Always at the end.
+ //
+ // Briefly, in 1.18, we considered the following automatic behavior, which for
+ // per-site output it would add it at the end of the expression and for per-sample
+ // output it would add it inside the square brackets:
+ // -f'%CHROM[ %SAMPLE]\n'
+ // -f'[%CHROM %SAMPLE\n]'
+ //
+ // However, this is an annoyance for users, as it is not entirely clear what
+ // will happen unless one understands the internals well (#1969)
+
+ register_tag(convert, "\n", 0, T_SEP);
}
int convert_set_option(convert_t *convert, enum convert_option opt, ...)
case subset_samples:
convert->subset_samples = va_arg(args, uint8_t**);
break;
+ case header_samples:
+ convert->header_samples = va_arg(args, int);
+ break;
+ case print_filtered:
+ convert->print_filtered = strdup(va_arg(args, char*));
+ break;
case force_newline:
convert->force_newline = va_arg(args, int);
if ( convert->force_newline ) force_newline_(convert);
break;
+ case no_hdr_indices:
+ convert->no_hdr_indices = va_arg(args, int);
+ break;
default:
ret = -1;
}
/* convert.c -- functions for converting between VCF/BCF and related formats.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <errno.h>
#include <sys/stat.h>
#include <sys/types.h>
+#include <stdint.h>
#define __STDC_FORMAT_MACROS
#include <inttypes.h>
#include <math.h>
#include <htslib/vcfutils.h>
#include <htslib/kfunc.h>
#include <htslib/khash_str2int.h>
+#include <htslib/hts_endian.h>
#include "bcftools.h"
#include "variantkey.h"
#include "convert.h"
char *undef_info_tag;
void *used_tags_hash;
char **used_tags_list;
+ char *print_filtered;
int nused_tags;
int allow_undef_tags;
int force_newline;
+ int header_samples;
+ int no_hdr_indices;
uint8_t **subset_samples;
};
}
else kputc('.', str);
}
-static inline int32_t bcf_array_ivalue(void *bcf_array, int type, int idx)
+static inline int32_t bcf_array_ivalue(uint8_t *bcf_array, int type, int idx)
{
if ( type==BCF_BT_INT8 )
{
- int8_t val = ((int8_t*)bcf_array)[idx];
+ int8_t val = le_to_i8(&bcf_array[idx * sizeof(val)]);
if ( val==bcf_int8_missing ) return bcf_int32_missing;
if ( val==bcf_int8_vector_end ) return bcf_int32_vector_end;
return val;
}
if ( type==BCF_BT_INT16 )
{
- int16_t val = ((int16_t*)bcf_array)[idx];
+ int16_t val = le_to_i16(&bcf_array[idx * sizeof(val)]);
if ( val==bcf_int16_missing ) return bcf_int32_missing;
if ( val==bcf_int16_vector_end ) return bcf_int32_vector_end;
return val;
}
- return ((int32_t*)bcf_array)[idx];
+ return le_to_i32(&bcf_array[idx * sizeof(int32_t)]);
}
static inline void _copy_field(char *src, uint32_t len, int idx, kstring_t *str)
{
kputc('.', str);
return;
}
- #define BRANCH(type_t, is_missing, is_vector_end, kprint) { \
- type_t val = ((type_t *) info->vptr)[fmt->subscript]; \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, kprint) { \
+ type_t val = convert(&info->vptr[fmt->subscript * sizeof(type_t)]); \
if ( is_missing || is_vector_end ) kputc('.',str); \
else kprint; \
}
switch (info->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, val==bcf_int8_missing, val==bcf_int8_vector_end, kputw(val, str)); break;
- case BCF_BT_INT16: BRANCH(int16_t, val==bcf_int16_missing, val==bcf_int16_vector_end, kputw(val, str)); break;
- case BCF_BT_INT32: BRANCH(int32_t, val==bcf_int32_missing, val==bcf_int32_vector_end, kputw(val, str)); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), kputd(val, str)); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, kputw(val, str)); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, kputw(val, str)); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, kputw(val, str)); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), kputd(val, str)); break;
case BCF_BT_CHAR: _copy_field((char*)info->vptr, info->vptr_len, fmt->subscript, str); break;
default: fprintf(bcftools_stderr,"todo: type %d\n", info->type); bcftools_exit(1); break;
}
}
if ( fmt->fmt->type == BCF_BT_FLOAT )
{
- float *ptr = (float*)(fmt->fmt->p + isample*fmt->fmt->size);
- if ( bcf_float_is_missing(ptr[fmt->subscript]) || bcf_float_is_vector_end(ptr[fmt->subscript]) )
+ uint8_t *ptr = fmt->fmt->p + isample*fmt->fmt->size;
+ float val = le_to_float(&ptr[fmt->subscript * sizeof(float)]);
+ if ( bcf_float_is_missing(val) || bcf_float_is_vector_end(val) )
kputc('.', str);
else
- kputd(ptr[fmt->subscript], str);
+ kputd(val, str);
}
else if ( fmt->fmt->type != BCF_BT_CHAR )
{
int mask = fmt->subscript==0 ? 3 : 1; // merge both haplotypes if subscript==0
- #define BRANCH(type_t, nbits) { \
- type_t *x = (type_t*)(fmt->fmt->p + isample*fmt->fmt->size); \
+ #define BRANCH(type_t, convert, nbits) { \
+ uint8_t *x = fmt->fmt->p + isample*fmt->fmt->size; \
int i,j; \
if ( fmt->subscript<=0 || fmt->subscript==1 ) \
{ \
for (j=0; j < fmt->fmt->n; j++) \
{ \
- type_t val = x[j]; \
+ type_t val = convert(&x[j * sizeof(type_t)]); \
if ( !val ) continue; \
for (i=0; i<nbits; i+=2) \
if ( val & (mask<<i) ) { kputs(csq->str[(j*30+i)/2], &csq->hap1); kputc_(',', &csq->hap1); } \
{ \
for (j=0; j < fmt->fmt->n; j++) \
{ \
- type_t val = x[j]; \
+ type_t val = convert(&x[j * sizeof(type_t)]); \
if ( !val ) continue; \
for (i=1; i<nbits; i+=2) \
if ( val & (1<<i) ) { kputs(csq->str[(j*30+i)/2], &csq->hap2); kputc_(',', &csq->hap2); } \
}
switch (fmt->fmt->type)
{
- case BCF_BT_INT8: BRANCH(uint8_t, 8); break;
- case BCF_BT_INT16: BRANCH(uint16_t,16); break;
- case BCF_BT_INT32: BRANCH(uint32_t,30); break; // 2 bytes unused to account for the reserved BCF values
+ case BCF_BT_INT8: BRANCH(uint8_t, le_to_u8, 8); break;
+ case BCF_BT_INT16: BRANCH(uint16_t, le_to_u16, 16); break;
+ case BCF_BT_INT32: BRANCH(uint32_t, le_to_u32, 30); break; // 2 bits unused to account for the reserved BCF values
default: error("Unexpected type: %d\n", fmt->fmt->type); bcftools_exit(1); break;
}
#undef BRANCH
int al = bcf_gt_allele(gt[i]);
if ( al > line->n_allele || al >= fmt->fmt->n ) goto invalid;
- #define BRANCH(type_t, missing, vector_end) { \
- type_t val = ((type_t *) fmt->fmt->p)[al + isample*fmt->fmt->n]; \
+ #define BRANCH(type_t, convert, missing, vector_end) { \
+ type_t val = convert(&fmt->fmt->p[(al + isample*fmt->fmt->n)*sizeof(type_t)]); \
if ( val==missing || val==vector_end ) goto invalid; \
else n[i] = val; \
}
switch (fmt->fmt->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_missing, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_missing, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_missing, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_missing, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_missing, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_missing, bcf_int32_vector_end); break;
default: goto invalid; break;
}
#undef BRANCH
if ( n[0]==n[1] ) kputc(n[0]==0 ? '.':'0', str);
else
{
- double pval = n[0] < n[1] ? kf_betai(n[1], n[0] + 1, 0.5) : kf_betai(n[0], n[1] + 1, 0.5);
- pval *= 2;
- if ( pval>=1 ) pval = 0; // this can happen, machine precision error, eg. kf_betai(1,0,0.5)
- else
- pval = -4.34294481903*log(pval);
+ double pval = calc_binom_two_sided(n[0],n[1],0.5);
+
+ // convrt to phred
+ if ( pval>=1 ) pval = 0;
+ else pval = -4.34294481903*log(pval);
kputd(pval, str);
}
return;
free(convert->used_tags_list);
}
khash_str2int_destroy(convert->used_tags_hash);
+ free(convert->print_filtered);
free(convert->fmt);
free(convert->undef_info_tag);
free(convert->dat);
int convert_header(convert_t *convert, kstring_t *str)
{
int i, icol = 0, l_ori = str->l;
+ bcf_hdr_t *hdr = convert->header;
- // Supress the header output if LINE is present
+ // Suppress the header output if LINE is present
for (i=0; i<convert->nfmt; i++)
if ( convert->fmt[i].type == T_LINE ) break;
if ( i!=convert->nfmt )
while ( convert->fmt[j].is_gt_field ) j++;
for (js=0; js<convert->nsamples; js++)
{
+ int ks = convert->samples[js];
for (k=i; k<j; k++)
{
if ( convert->fmt[k].type == T_SEP )
}
}
}
+ else if ( convert->header_samples )
+ {
+ icol++;
+ if ( !convert->no_hdr_indices ) ksprintf(str,"[%d]",icol);
+ ksprintf(str,"%s:%s", hdr->samples[ks], convert->fmt[k].key);
+ }
else
- ksprintf(str, "[%d]%s", ++icol, convert->fmt[k].key);
+ {
+ icol++;
+ if ( !convert->no_hdr_indices ) ksprintf(str,"[%d]",icol);
+ ksprintf(str,"%s", convert->fmt[k].key);
+ }
+ }
+ if ( has_fmt_newline )
+ {
+ if ( !convert->header_samples ) break;
+
+ // this is unfortunate: the formatting expression breaks the per-sample output into separate lines,
+ // therefore including a sample name in the header makes no sense anymore
+ convert->header_samples = 0;
+ str->l = l_ori;
+ return convert_header(convert, str);
}
- if ( has_fmt_newline ) break;
}
i = j-1;
continue;
if ( convert->fmt[i].key ) kputs(convert->fmt[i].key, str);
continue;
}
- ksprintf(str, "[%d]%s", ++icol, convert->fmt[i].key);
+ icol++;
+ if ( !convert->no_hdr_indices ) ksprintf(str,"[%d]",icol);
+ ksprintf(str,"%s", convert->fmt[i].key);
}
if ( has_fmt_newline ) kputc('\n',str);
return str->l - l_ori;
{
// Skip samples when filtering was requested
int ks = convert->samples[js];
- if ( convert->subset_samples && *convert->subset_samples && !(*convert->subset_samples)[ks] ) continue;
+ if ( convert->subset_samples && *convert->subset_samples && !(*convert->subset_samples)[ks] )
+ {
+ if ( !convert->print_filtered ) continue;
+
+ for (k=i; k<j; k++)
+ if ( convert->fmt[k].type==T_SEP )
+ convert->fmt[k].handler(convert, line, &convert->fmt[k], ks, str);
+ else
+ kputs(convert->print_filtered, str);
+ continue;
+ }
// Here comes a hack designed for TBCSQ. When running on large files,
// such as 1000GP, there are too many empty fields in the output and
}
if ( has_newline ) return;
- // A newline is not present, force it. But where to add it?
- // Consider
- // -f'%CHROM[ %SAMPLE]\n'
- // vs
- // -f'[%CHROM %SAMPLE\n]'
- for (i=0; i<convert->nfmt; i++)
- if ( !convert->fmt[i].is_gt_field && convert->fmt[i].key ) break;
-
- if ( i < convert->nfmt )
- register_tag(convert, "\n", 0, T_SEP); // the first case
- else
- {
- // the second case
- i = convert->nfmt - 1;
- if ( !convert->fmt[i].key )
- {
- convert->fmt[i].key = strdup("\n");
- convert->fmt[i].is_gt_field = 1;
- register_tag(convert, NULL, 0, T_SEP);
- }
- else
- register_tag(convert, "\n", 1, T_SEP);
- }
+ // A newline is not present, force it. But where to add it? Always at the end.
+ //
+ // Briefly, in 1.18, we considered the following automatic behavior, which for
+ // per-site output it would add it at the end of the expression and for per-sample
+ // output it would add it inside the square brackets:
+ // -f'%CHROM[ %SAMPLE]\n'
+ // -f'[%CHROM %SAMPLE\n]'
+ //
+ // However, this is an annoyance for users, as it is not entirely clear what
+ // will happen unless one understands the internals well (#1969)
+
+ register_tag(convert, "\n", 0, T_SEP);
}
int convert_set_option(convert_t *convert, enum convert_option opt, ...)
case subset_samples:
convert->subset_samples = va_arg(args, uint8_t**);
break;
+ case header_samples:
+ convert->header_samples = va_arg(args, int);
+ break;
+ case print_filtered:
+ convert->print_filtered = strdup(va_arg(args, char*));
+ break;
case force_newline:
convert->force_newline = va_arg(args, int);
if ( convert->force_newline ) force_newline_(convert);
break;
+ case no_hdr_indices:
+ convert->no_hdr_indices = va_arg(args, int);
+ break;
default:
ret = -1;
}
/* convert.h -- functions for converting between VCF/BCF and related formats.
- Copyright (C) 2014-2023 Genome Research Ltd.
+ Copyright (C) 2014-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
typedef struct _convert_t convert_t;
enum convert_option
{
- allow_undef_tags,
- subset_samples,
- force_newline,
+ allow_undef_tags, // see `bcftools query --allow-undef-tags`, throws an error if tag is not defined otherwise
+ subset_samples, // in bracketed expressions (e.g. [ %GT]) consider only marked samples
+ header_samples, // include sample name in bracketed tags (e.g. SAMPLE1:GT SAMPLE2:GT for [ %GT])
+ force_newline, // automatically insert a newline when not part of the formatting expression
+ print_filtered, // print the provided string instead of discarding samples not included in subset_samples
+ no_hdr_indices, // drop column indices when printing header, i.e. "#CHROM", not "#[1]CHROM"
};
convert_t *convert_init(bcf_hdr_t *hdr, int *samples, int nsamples, const char *str);
/* The MIT License
- Copyright (c) 2016-2023 Genome Research Ltd.
+ Copyright (c) 2016-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
Read about transcript types here
http://vega.sanger.ac.uk/info/about/gene_and_transcript_types.html
- http://www.ensembl.org/info/genome/variation/predicted_data.html
+ https://www.ensembl.org/info/genome/variation/prediction/predicted_data.html
https://www.gencodegenes.org/pages/biotypes.html
List of supported biotypes
#include <errno.h>
#include <unistd.h>
#include <ctype.h>
+#include <strings.h>
#include "bcftools.h"
#include "filter.h"
#include "regidx.h"
if ( args->hdr_nsmpl )
bcf_hdr_printf(args->hdr,"##FORMAT=<ID=%s,Number=.,Type=Integer,Description=\"Bitmask of indexes to INFO/BCSQ, with interleaved first/second haplotype. Use \\\"bcftools query -f'[%%CHROM\\t%%POS\\t%%SAMPLE\\t%%TBCSQ\\n]'\\\" to translate.\">",args->bcsq_tag);
if ( bcf_hdr_write(args->out_fh, args->hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname?args->output_fname:"standard output");
- if ( args->write_index && init_index(args->out_fh,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr,args->output_fname,
+ &args->index_fn, args->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
}
if ( args->verbosity > 0 ) fprintf(stderr,"Calling...\n");
}
}
/*
- The splice_* functions are for consquences around splice sites: start,stop,splice_*
+ The splice_* functions are for consequences around splice sites: start,stop,splice_*
*/
#define SPLICE_VAR_REF 0 // ref: ACGT>ACGT, csq not applicable, skip completely
#define SPLICE_OUTSIDE 1 // splice acceptor or similar; csq set and is done, does not overlap the region
static inline void csq_stage_splice(args_t *args, bcf1_t *rec, gf_tscript_t *tr, uint32_t type, int ial)
{
#if XDBG
-fprintf(stderr,"csq_stage_splice %d: type=%d\n",rec->pos+1,type);
+fprintf(stderr,"csq_stage_splice %d: type=%d\n",(int)rec->pos+1,type);
#endif
if ( !type ) return;
csq_t csq;
if ( tr->strand==STRAND_FWD && splice->vcf.pos >= ex_beg + 3 ) return 0;
#if XDBG
- fprintf(stderr,"shifted_del_synonymous: %d-%d %s\n",ex_beg,ex_end, tr->strand==STRAND_FWD?"fwd":"rev");
+ fprintf(stderr,"shifted_del_synonymous: %d-%d %s\n",ex_beg,ex_end, tr->strand==STRAND_FWD?"fwd":(tr->strand==STRAND_REV?"rev":"unk"));
fprintf(stderr," %d .. %s > %s\n",splice->vcf.pos+1,splice->vcf.ref,splice->vcf.alt);
#endif
while ( ptr_vcf[i] && ptr_vcf[i]==ptr_ref[i] ) i++;
if ( ptr_vcf[i] ) return 0; // the deleted sequence cannot be replaced
}
- else
+ else if ( tr->strand==STRAND_FWD )
{
// STRAND_FWD
int32_t vcf_block_beg = splice->vcf.pos + ref_len - 2*ndel; // the position of the first base of the ref block that could potentially replace the deletion
splice->kalt.l = 0; kputsn(splice->vcf.alt + splice->tbeg, splice->vcf.alen, &splice->kalt);
if ( (splice->ref_beg+1 < ex_beg && splice->ref_end >= ex_beg) || (splice->ref_beg+1 < ex_end && splice->ref_end >= ex_end) ) // ouch, ugly ENST00000409523/long-overlapping-del.vcf
{
- splice->csq |= (splice->ref_end - splice->ref_beg)%3 ? CSQ_FRAMESHIFT_VARIANT : CSQ_INFRAME_DELETION;
+ int ref_beg = splice->ref_beg + splice->kalt.l - 1; // 0 for AAA>A, 1 for AAA>AC
+ if ( ref_beg < splice->ref_end )
+ splice->csq |= (splice->ref_end - ref_beg)%3 ? CSQ_FRAMESHIFT_VARIANT : CSQ_INFRAME_DELETION;
return SPLICE_OVERLAP;
}
}
{
if ( splice->check_region_beg ) splice->csq |= CSQ_SPLICE_REGION;
if ( splice->tr->strand==STRAND_FWD ) { if ( splice->check_start ) splice->csq |= CSQ_START_LOST; }
- else { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
+ else if ( splice->tr->strand==STRAND_REV ) { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
}
if ( splice->ref_end > ex_end - 3 )
{
if ( splice->check_region_end ) splice->csq |= CSQ_SPLICE_REGION;
if ( splice->tr->strand==STRAND_REV ) { if ( splice->check_start ) splice->csq |= CSQ_START_LOST; }
- else { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
+ else if ( splice->tr->strand==STRAND_FWD ) { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
}
if ( splice->set_refalt )
{
if ( !(tr->trim & TRIM_5PRIME) )
{
if ( tr->strand==STRAND_FWD ) { if ( child->icds==0 ) splice.check_start = 1; }
- else { if ( child->icds==tr->ncds-1 ) splice.check_start = 1; }
+ else if ( tr->strand==STRAND_REV ) { if ( child->icds==tr->ncds-1 ) splice.check_start = 1; }
}
if ( !(tr->trim & TRIM_3PRIME) )
{
if ( tr->strand==STRAND_FWD ) { if ( child->icds==tr->ncds-1 ) splice.check_stop = 1; }
- else { if ( child->icds==0 ) splice.check_stop = 1; }
+ else if ( tr->strand==STRAND_REV ) { if ( child->icds==0 ) splice.check_stop = 1; }
}
if ( splice.check_start ) // do not check starts in incomplete CDS, defined as not starting with M
{
if ( tr->strand==STRAND_FWD ) { if ( dna2aa(TSCRIPT_AUX(tr)->ref+N_REF_PAD+cds->beg-tr->beg) != 'M' ) splice.check_start = 0; }
- else { if ( cdna2aa(TSCRIPT_AUX(tr)->ref+N_REF_PAD+cds->beg-tr->beg+cds->len-3) != 'M' ) splice.check_start = 0; }
+ else if ( tr->strand==STRAND_REV ) { if ( cdna2aa(TSCRIPT_AUX(tr)->ref+N_REF_PAD+cds->beg-tr->beg+cds->len-3) != 'M' ) splice.check_start = 0; }
}
if ( child->icds!=0 ) splice.check_region_beg = 1;
if ( child->icds!=tr->ncds-1 ) splice.check_region_end = 1;
}
}
}
- else // STRAND_REV
+ else if ( strand==STRAND_REV )
{
// right padding - number of bases to take from ref
npad = (seq.m - (sbeg + seq.l)) % 3;
}
}
}
+ else error("Should not happen: %d\n", strand);
kputc_(0,tseq); tseq->l--;
#if DBG
fprintf(stderr," tseq: %s\n", tseq->s);
int csq_push(args_t *args, csq_t *csq, bcf1_t *rec)
{
#if XDBG
-fprintf(stderr,"csq_push: %d .. %d\n",rec->pos+1,csq->type.type);
+fprintf(stderr,"csq_push: %d .. %d\n",(int)rec->pos+1,csq->type.type);
#endif
khint_t k = kh_get(pos2vbuf, args->pos2vbuf, (int)csq->pos);
vbuf_t *vbuf = (k == kh_end(args->pos2vbuf)) ? NULL : kh_val(args->pos2vbuf, k);
kputs(gf_type2gff_string(csq->biotype), str);
if ( CSQ_PRN_STRAND(csq->type) || csq->vstr.l )
- kputs(csq->strand==STRAND_FWD ? "|+" : "|-", str);
+ kputs(csq->strand==STRAND_FWD ? "|+" : (csq->strand==STRAND_REV ? "|-" : "|."), str);
if ( csq->vstr.l )
kputs(csq->vstr.s, str);
{
int i;
gf_tscript_t *tr = hap->tr;
+ assert( tr->strand==STRAND_FWD || tr->strand==STRAND_REV );
int ref_node = tr->strand==STRAND_FWD ? ibeg : iend;
int icsq = node->ncsq_list++;
hts_expand0(csq_t,node->ncsq_list,node->mcsq_list,node->csq_list);
indel = 0;
}
}
- else
+ else if ( tr->strand==STRAND_REV )
{
i = istack + 1, ibeg = -1;
while ( --i > 0 )
" --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n"
" --threads INT Use multithreading with <int> worker threads [0]\n"
" -v, --verbose INT Verbosity level 0-2 [1]\n"
- " --write-index Automatically index the output files [off]\n"
+ " -W, --write-index[=FMT] Automatically index the output files [off]\n"
"\n"
"Example:\n"
- " bcftools csq -f hs37d5.fa -g Homo_sapiens.GRCh37.82.gff3.gz in.vcf\n"
+ " bcftools csq -f hs37d5.fa -g Homo_sapiens.GRCh37.87.gff3.gz in.vcf\n"
"\n"
" # GFF3 annotation files can be downloaded from Ensembl. e.g. for human:\n"
- " ftp://ftp.ensembl.org/pub/current_gff3/homo_sapiens/\n"
- " ftp://ftp.ensembl.org/pub/grch37/release-84/gff3/homo_sapiens/\n"
+ " http://ftp.ensembl.org/pub/current_gff3/homo_sapiens/\n"
+ " http://ftp.ensembl.org/pub/grch37/current/gff3/homo_sapiens/\n"
"\n";
}
{"targets-file",1,0,'T'},
{"targets-overlap",required_argument,NULL,5},
{"no-version",no_argument,NULL,3},
- {"write-index",no_argument,NULL,6},
+ {"write-index",optional_argument,NULL,'W'},
{"dump-gff",required_argument,NULL,7},
{"unify-chr-names",required_argument,NULL,8},
{0,0,0,0}
int regions_overlap = 1;
int targets_overlap = 0;
char *targets_list = NULL, *regions_list = NULL, *tmp;
- while ((c = getopt_long(argc, argv, "?hr:R:t:T:i:e:f:o:O:g:s:S:p:qc:ln:bB:v:",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "?hr:R:t:T:i:e:f:o:O:g:s:S:p:qc:ln:bB:v:W::",loptions,NULL)) >= 0)
{
switch (c)
{
targets_overlap = parse_overlap_option(optarg);
if ( targets_overlap < 0 ) error("Could not parse: --targets-overlap %s\n",optarg);
break;
- case 6 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 7 : args->dump_gff = optarg; break;
case 8 :
if ( !strcmp(optarg,"0") ) args->unify_chr_names = 0;
}
else fname = argv[optind];
if ( argc - optind>1 ) error("%s", usage());
- if ( !args->fa_fname ) error("Missing the --fa-ref option\n");
+ if ( !args->fa_fname ) error("Missing the --fasta-ref option\n");
if ( !args->gff_fname ) error("Missing the --gff option\n");
args->sr = bcf_sr_init();
if ( targets_list )
/* The MIT License
- Copyright (c) 2016-2023 Genome Research Ltd.
+ Copyright (c) 2016-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
Read about transcript types here
http://vega.sanger.ac.uk/info/about/gene_and_transcript_types.html
- http://www.ensembl.org/info/genome/variation/predicted_data.html
+ https://www.ensembl.org/info/genome/variation/prediction/predicted_data.html
https://www.gencodegenes.org/pages/biotypes.html
List of supported biotypes
#include <errno.h>
#include <unistd.h>
#include <ctype.h>
+#include <strings.h>
#include "bcftools.h"
#include "filter.h"
#include "regidx.h"
if ( args->hdr_nsmpl )
bcf_hdr_printf(args->hdr,"##FORMAT=<ID=%s,Number=.,Type=Integer,Description=\"Bitmask of indexes to INFO/BCSQ, with interleaved first/second haplotype. Use \\\"bcftools query -f'[%%CHROM\\t%%POS\\t%%SAMPLE\\t%%TBCSQ\\n]'\\\" to translate.\">",args->bcsq_tag);
if ( bcf_hdr_write(args->out_fh, args->hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname?args->output_fname:"standard output");
- if ( args->write_index && init_index(args->out_fh,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr,args->output_fname,
+ &args->index_fn, args->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
}
if ( args->verbosity > 0 ) fprintf(bcftools_stderr,"Calling...\n");
}
}
/*
- The splice_* functions are for consquences around splice sites: start,stop,splice_*
+ The splice_* functions are for consequences around splice sites: start,stop,splice_*
*/
#define SPLICE_VAR_REF 0 // ref: ACGT>ACGT, csq not applicable, skip completely
#define SPLICE_OUTSIDE 1 // splice acceptor or similar; csq set and is done, does not overlap the region
static inline void csq_stage_splice(args_t *args, bcf1_t *rec, gf_tscript_t *tr, uint32_t type, int ial)
{
#if XDBG
-fprintf(bcftools_stderr,"csq_stage_splice %d: type=%d\n",rec->pos+1,type);
+fprintf(bcftools_stderr,"csq_stage_splice %d: type=%d\n",(int)rec->pos+1,type);
#endif
if ( !type ) return;
csq_t csq;
if ( tr->strand==STRAND_FWD && splice->vcf.pos >= ex_beg + 3 ) return 0;
#if XDBG
- fprintf(bcftools_stderr,"shifted_del_synonymous: %d-%d %s\n",ex_beg,ex_end, tr->strand==STRAND_FWD?"fwd":"rev");
+ fprintf(bcftools_stderr,"shifted_del_synonymous: %d-%d %s\n",ex_beg,ex_end, tr->strand==STRAND_FWD?"fwd":(tr->strand==STRAND_REV?"rev":"unk"));
fprintf(bcftools_stderr," %d .. %s > %s\n",splice->vcf.pos+1,splice->vcf.ref,splice->vcf.alt);
#endif
while ( ptr_vcf[i] && ptr_vcf[i]==ptr_ref[i] ) i++;
if ( ptr_vcf[i] ) return 0; // the deleted sequence cannot be replaced
}
- else
+ else if ( tr->strand==STRAND_FWD )
{
// STRAND_FWD
int32_t vcf_block_beg = splice->vcf.pos + ref_len - 2*ndel; // the position of the first base of the ref block that could potentially replace the deletion
splice->kalt.l = 0; kputsn(splice->vcf.alt + splice->tbeg, splice->vcf.alen, &splice->kalt);
if ( (splice->ref_beg+1 < ex_beg && splice->ref_end >= ex_beg) || (splice->ref_beg+1 < ex_end && splice->ref_end >= ex_end) ) // ouch, ugly ENST00000409523/long-overlapping-del.vcf
{
- splice->csq |= (splice->ref_end - splice->ref_beg)%3 ? CSQ_FRAMESHIFT_VARIANT : CSQ_INFRAME_DELETION;
+ int ref_beg = splice->ref_beg + splice->kalt.l - 1; // 0 for AAA>A, 1 for AAA>AC
+ if ( ref_beg < splice->ref_end )
+ splice->csq |= (splice->ref_end - ref_beg)%3 ? CSQ_FRAMESHIFT_VARIANT : CSQ_INFRAME_DELETION;
return SPLICE_OVERLAP;
}
}
{
if ( splice->check_region_beg ) splice->csq |= CSQ_SPLICE_REGION;
if ( splice->tr->strand==STRAND_FWD ) { if ( splice->check_start ) splice->csq |= CSQ_START_LOST; }
- else { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
+ else if ( splice->tr->strand==STRAND_REV ) { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
}
if ( splice->ref_end > ex_end - 3 )
{
if ( splice->check_region_end ) splice->csq |= CSQ_SPLICE_REGION;
if ( splice->tr->strand==STRAND_REV ) { if ( splice->check_start ) splice->csq |= CSQ_START_LOST; }
- else { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
+ else if ( splice->tr->strand==STRAND_FWD ) { if ( splice->check_stop ) splice->csq |= CSQ_STOP_LOST; }
}
if ( splice->set_refalt )
{
if ( !(tr->trim & TRIM_5PRIME) )
{
if ( tr->strand==STRAND_FWD ) { if ( child->icds==0 ) splice.check_start = 1; }
- else { if ( child->icds==tr->ncds-1 ) splice.check_start = 1; }
+ else if ( tr->strand==STRAND_REV ) { if ( child->icds==tr->ncds-1 ) splice.check_start = 1; }
}
if ( !(tr->trim & TRIM_3PRIME) )
{
if ( tr->strand==STRAND_FWD ) { if ( child->icds==tr->ncds-1 ) splice.check_stop = 1; }
- else { if ( child->icds==0 ) splice.check_stop = 1; }
+ else if ( tr->strand==STRAND_REV ) { if ( child->icds==0 ) splice.check_stop = 1; }
}
if ( splice.check_start ) // do not check starts in incomplete CDS, defined as not starting with M
{
if ( tr->strand==STRAND_FWD ) { if ( dna2aa(TSCRIPT_AUX(tr)->ref+N_REF_PAD+cds->beg-tr->beg) != 'M' ) splice.check_start = 0; }
- else { if ( cdna2aa(TSCRIPT_AUX(tr)->ref+N_REF_PAD+cds->beg-tr->beg+cds->len-3) != 'M' ) splice.check_start = 0; }
+ else if ( tr->strand==STRAND_REV ) { if ( cdna2aa(TSCRIPT_AUX(tr)->ref+N_REF_PAD+cds->beg-tr->beg+cds->len-3) != 'M' ) splice.check_start = 0; }
}
if ( child->icds!=0 ) splice.check_region_beg = 1;
if ( child->icds!=tr->ncds-1 ) splice.check_region_end = 1;
}
}
}
- else // STRAND_REV
+ else if ( strand==STRAND_REV )
{
// right padding - number of bases to take from ref
npad = (seq.m - (sbeg + seq.l)) % 3;
}
}
}
+ else error("Should not happen: %d\n", strand);
kputc_(0,tseq); tseq->l--;
#if DBG
fprintf(bcftools_stderr," tseq: %s\n", tseq->s);
int csq_push(args_t *args, csq_t *csq, bcf1_t *rec)
{
#if XDBG
-fprintf(bcftools_stderr,"csq_push: %d .. %d\n",rec->pos+1,csq->type.type);
+fprintf(bcftools_stderr,"csq_push: %d .. %d\n",(int)rec->pos+1,csq->type.type);
#endif
khint_t k = kh_get(pos2vbuf, args->pos2vbuf, (int)csq->pos);
vbuf_t *vbuf = (k == kh_end(args->pos2vbuf)) ? NULL : kh_val(args->pos2vbuf, k);
kputs(gf_type2gff_string(csq->biotype), str);
if ( CSQ_PRN_STRAND(csq->type) || csq->vstr.l )
- kputs(csq->strand==STRAND_FWD ? "|+" : "|-", str);
+ kputs(csq->strand==STRAND_FWD ? "|+" : (csq->strand==STRAND_REV ? "|-" : "|."), str);
if ( csq->vstr.l )
kputs(csq->vstr.s, str);
{
int i;
gf_tscript_t *tr = hap->tr;
+ assert( tr->strand==STRAND_FWD || tr->strand==STRAND_REV );
int ref_node = tr->strand==STRAND_FWD ? ibeg : iend;
int icsq = node->ncsq_list++;
hts_expand0(csq_t,node->ncsq_list,node->mcsq_list,node->csq_list);
indel = 0;
}
}
- else
+ else if ( tr->strand==STRAND_REV )
{
i = istack + 1, ibeg = -1;
while ( --i > 0 )
" --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n"
" --threads INT Use multithreading with <int> worker threads [0]\n"
" -v, --verbose INT Verbosity level 0-2 [1]\n"
- " --write-index Automatically index the output files [off]\n"
+ " -W, --write-index[=FMT] Automatically index the output files [off]\n"
"\n"
"Example:\n"
- " bcftools csq -f hs37d5.fa -g Homo_sapiens.GRCh37.82.gff3.gz in.vcf\n"
+ " bcftools csq -f hs37d5.fa -g Homo_sapiens.GRCh37.87.gff3.gz in.vcf\n"
"\n"
" # GFF3 annotation files can be downloaded from Ensembl. e.g. for human:\n"
- " ftp://ftp.ensembl.org/pub/current_gff3/homo_sapiens/\n"
- " ftp://ftp.ensembl.org/pub/grch37/release-84/gff3/homo_sapiens/\n"
+ " http://ftp.ensembl.org/pub/current_gff3/homo_sapiens/\n"
+ " http://ftp.ensembl.org/pub/grch37/current/gff3/homo_sapiens/\n"
"\n";
}
{"targets-file",1,0,'T'},
{"targets-overlap",required_argument,NULL,5},
{"no-version",no_argument,NULL,3},
- {"write-index",no_argument,NULL,6},
+ {"write-index",optional_argument,NULL,'W'},
{"dump-gff",required_argument,NULL,7},
{"unify-chr-names",required_argument,NULL,8},
{0,0,0,0}
int regions_overlap = 1;
int targets_overlap = 0;
char *targets_list = NULL, *regions_list = NULL, *tmp;
- while ((c = getopt_long(argc, argv, "?hr:R:t:T:i:e:f:o:O:g:s:S:p:qc:ln:bB:v:",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "?hr:R:t:T:i:e:f:o:O:g:s:S:p:qc:ln:bB:v:W::",loptions,NULL)) >= 0)
{
switch (c)
{
targets_overlap = parse_overlap_option(optarg);
if ( targets_overlap < 0 ) error("Could not parse: --targets-overlap %s\n",optarg);
break;
- case 6 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 7 : args->dump_gff = optarg; break;
case 8 :
if ( !strcmp(optarg,"0") ) args->unify_chr_names = 0;
}
else fname = argv[optind];
if ( argc - optind>1 ) error("%s", usage());
- if ( !args->fa_fname ) error("Missing the --fa-ref option\n");
+ if ( !args->fa_fname ) error("Missing the --fasta-ref option\n");
if ( !args->gff_fname ) error("Missing the --gff option\n");
args->sr = bcf_sr_init();
if ( targets_list )
--- /dev/null
+/*
+ * A cut down C translated of the C++ edlib.cpp file.
+ * Taken from edlib v0.1.0-166-g931be2b
+ */
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdbool.h>
+
+#include "edlib.h"
+
+typedef uint64_t Word;
+static const int WORD_SIZE = 64; // Size of Word in bits
+static const Word WORD_1 = (Word)1;
+static const Word HIGH_BIT_MASK = 1LL << 63; // 100..00
+//#define MAX_UCHAR 255
+#define MAX_UCHAR 7 // better cache usage for our data
+
+#ifndef MAX
+#define MAX(a,b) ((a)>(b)?(a):(b))
+#endif
+
+typedef struct Block {
+ Word P; // Pvin
+ Word M; // Mvin
+ int score; // score of last cell in block;
+} Block;
+
+
+/**
+ * Defines equality relation on alphabet characters.
+ * By default each character is always equal only to itself, but you can also provide additional equalities.
+ */
+typedef struct EqualityDefinition {
+ bool matrix[MAX_UCHAR + 1][MAX_UCHAR + 1];
+} EqualityDefinition;
+
+static EqualityDefinition *
+CreateEqualityDefinition(const char *alphabet, int alphabet_size,
+ const EdlibEqualityPair* additionalEqualities,
+ const int additionalEqualitiesLength) {
+ EqualityDefinition *ed = malloc(sizeof(*ed));
+
+ for (size_t i = 0; i < alphabet_size; i++) {
+ for (size_t j = 0; j < alphabet_size; j++) {
+ ed->matrix[i][j] = (i == j);
+ }
+ }
+ if (additionalEqualities != NULL) {
+ for (int i = 0; i < additionalEqualitiesLength; i++) {
+ const char *firstTransformed = strchr(alphabet, additionalEqualities[i].first);
+ const char *secondTransformed = strchr(alphabet, additionalEqualities[i].second);
+ if (firstTransformed && alphabet_size) {
+ ed->matrix[firstTransformed - alphabet][secondTransformed - alphabet] =
+ ed->matrix[secondTransformed - alphabet][firstTransformed - alphabet]
+ = true;
+ }
+ }
+ }
+
+ return ed;
+}
+
+/**
+ * @param a Element from transformed sequence.
+ * @param b Element from transformed sequence.
+ * @return True if a and b are defined as equal, false otherwise.
+ */
+static inline const /* attribute pure or const? */
+bool equalityDefinition_areEqual(const EqualityDefinition *ed, unsigned char a, unsigned char b) {
+ return ed->matrix[a][b];
+}
+
+static int myersCalcEditDistanceSemiGlobal(const Word* Peq, int W, int maxNumBlocks,
+ int queryLength,
+ const unsigned char* target, int targetLength,
+ int k, EdlibAlignMode mode,
+ int* bestScore_, int** positions_, int* numPositions_);
+
+static char *transformSequences(const char* queryOriginal, int queryLength,
+ const char* targetOriginal, int targetLength,
+ unsigned char** queryTransformed,
+ unsigned char** targetTransformed,
+ int *alphabet_size);
+
+static inline int ceilDiv(int x, int y);
+
+static inline unsigned char* createReverseCopy(const unsigned char* seq, int length);
+
+static inline Word* buildPeq(const int alphabetLength,
+ const unsigned char* query,
+ const int queryLength,
+ const EqualityDefinition* equalityDefinition);
+
+
+/**
+ * Main edlib method.
+ */
+EdlibAlignResult edlibAlign(const char* const queryOriginal, const int queryLength,
+ const char* const targetOriginal, const int targetLength,
+ const EdlibAlignConfig config) {
+ EdlibAlignResult result;
+ result.status = EDLIB_STATUS_OK;
+ result.editDistance = -1;
+ result.endLocations = result.startLocations = NULL;
+ result.numLocations = 0;
+ result.alignment = NULL;
+ result.alignmentLength = 0;
+ result.alphabetLength = 0;
+
+ /*------------ TRANSFORM SEQUENCES AND RECOGNIZE ALPHABET -----------*/
+ unsigned char* query, * target;
+ int alphabet_size;
+ char *alphabet = transformSequences(queryOriginal, queryLength, targetOriginal, targetLength,
+ &query, &target, &alphabet_size);
+ result.alphabetLength = alphabet_size;
+ /*-------------------------------------------------------*/
+
+ // Handle special situation when at least one of the sequences has length 0.
+ if (queryLength == 0 || targetLength == 0) {
+ if (config.mode == EDLIB_MODE_NW) {
+ result.editDistance = MAX(queryLength, targetLength);
+ result.endLocations = malloc(sizeof(int) * 1);
+ result.endLocations[0] = targetLength - 1;
+ result.numLocations = 1;
+ } else if (config.mode == EDLIB_MODE_SHW || config.mode == EDLIB_MODE_HW) {
+ result.editDistance = queryLength;
+ result.endLocations = malloc(sizeof(int) * 1);
+ result.endLocations[0] = -1;
+ result.numLocations = 1;
+ } else {
+ result.status = EDLIB_STATUS_ERROR;
+ }
+
+ free(query);
+ free(target);
+ free(alphabet);
+ return result;
+ }
+
+ /*--------------------- INITIALIZATION ------------------*/
+ int maxNumBlocks = ceilDiv(queryLength, WORD_SIZE); // bmax in Myers
+ int W = maxNumBlocks * WORD_SIZE - queryLength; // number of redundant cells in last level blocks
+ EqualityDefinition *equalityDefinition =
+ CreateEqualityDefinition(alphabet, alphabet_size, config.additionalEqualities, config.additionalEqualitiesLength);
+ Word* Peq = buildPeq(alphabet_size, query, queryLength, equalityDefinition);
+ /*-------------------------------------------------------*/
+
+ /*------------------ MAIN CALCULATION -------------------*/
+ // TODO: Store alignment data only after k is determined? That could make things faster.
+// int positionNW; // Used only when mode is NW.
+// AlignmentData* alignData = NULL;
+ bool dynamicK = false;
+ int k = config.k;
+ if (k < 0) { // If valid k is not given, auto-adjust k until solution is found.
+ dynamicK = true;
+ k = WORD_SIZE; // Gives better results than smaller k.
+ }
+
+ do {
+ if (config.mode == EDLIB_MODE_HW || config.mode == EDLIB_MODE_SHW) {
+ myersCalcEditDistanceSemiGlobal(Peq, W, maxNumBlocks,
+ queryLength, target, targetLength,
+ k, config.mode, &(result.editDistance),
+ &(result.endLocations), &(result.numLocations));
+ } else { // mode == EDLIB_MODE_NW
+// myersCalcEditDistanceNW(Peq, W, maxNumBlocks,
+// queryLength, target, targetLength,
+// k, &(result.editDistance), &positionNW,
+// false, &alignData, -1);
+ }
+ k *= 2;
+ } while(dynamicK && result.editDistance == -1);
+
+ if (result.editDistance >= 0) { // If there is solution.
+ // If NW mode, set end location explicitly.
+ if (config.mode == EDLIB_MODE_NW) {
+ result.endLocations = malloc(sizeof(int) * 1);
+ result.endLocations[0] = targetLength - 1;
+ result.numLocations = 1;
+ }
+
+ // Find starting locations.
+ if (config.task == EDLIB_TASK_LOC || config.task == EDLIB_TASK_PATH) {
+ result.startLocations = malloc(result.numLocations * sizeof(int));
+ if (config.mode == EDLIB_MODE_HW) { // If HW, I need to calculate start locations.
+ const unsigned char* rTarget = createReverseCopy(target, targetLength);
+ const unsigned char* rQuery = createReverseCopy(query, queryLength);
+ // Peq for reversed query.
+ Word* rPeq = buildPeq(alphabet_size, rQuery, queryLength, equalityDefinition);
+ for (int i = 0; i < result.numLocations; i++) {
+ int endLocation = result.endLocations[i];
+ if (endLocation == -1) {
+ // NOTE: Sometimes one of optimal solutions is that query starts before target, like this:
+ // AAGG <- target
+ // CCTT <- query
+ // It will never be only optimal solution and it does not happen often, however it is
+ // possible and in that case end location will be -1. What should we do with that?
+ // Should we just skip reporting such end location, although it is a solution?
+ // If we do report it, what is the start location? -4? -1? Nothing?
+ // TODO: Figure this out. This has to do in general with how we think about start
+ // and end locations.
+ // Also, we have alignment later relying on this locations to limit the space of it's
+ // search -> how can it do it right if these locations are negative or incorrect?
+ result.startLocations[i] = 0; // I put 0 for now, but it does not make much sense.
+ } else {
+ int bestScoreSHW, numPositionsSHW;
+ int* positionsSHW;
+ myersCalcEditDistanceSemiGlobal(
+ rPeq, W, maxNumBlocks,
+ queryLength, rTarget + targetLength - endLocation - 1, endLocation + 1,
+ result.editDistance, EDLIB_MODE_SHW,
+ &bestScoreSHW, &positionsSHW, &numPositionsSHW);
+ // Taking last location as start ensures that alignment will not start with insertions
+ // if it can start with mismatches instead.
+ result.startLocations[i] = endLocation - positionsSHW[numPositionsSHW - 1];
+ free(positionsSHW);
+ }
+ }
+ free((void *)rTarget);
+ free((void *)rQuery);
+ free(rPeq);
+ } else { // If mode is SHW or NW
+ for (int i = 0; i < result.numLocations; i++) {
+ result.startLocations[i] = 0;
+ }
+ }
+ }
+ }
+ /*-------------------------------------------------------*/
+
+ //--- Free memory ---//
+ free(Peq);
+ free(query);
+ free(target);
+ free(alphabet);
+ free(equalityDefinition);
+// DestroyAlignmentData(alignData);
+ //-------------------//
+
+ return result;
+}
+
+/**
+ * Build Peq table for given query and alphabet.
+ * Peq is table of dimensions alphabetLength+1 x maxNumBlocks.
+ * Bit i of Peq[s * maxNumBlocks + b] is 1 if i-th symbol from block b of query equals symbol s, otherwise it is 0.
+ * NOTICE: free returned array with free()!
+ */
+static inline Word* buildPeq(const int alphabetLength,
+ const unsigned char* const query,
+ const int queryLength,
+ const EqualityDefinition* equalityDefinition) {
+ int maxNumBlocks = ceilDiv(queryLength, WORD_SIZE);
+ // table of dimensions alphabetLength+1 x maxNumBlocks. Last symbol is wildcard.
+ Word* Peq = malloc((alphabetLength + 1) * maxNumBlocks * sizeof(*Peq));
+
+ // Build Peq (1 is match, 0 is mismatch). NOTE: last column is wildcard(symbol that matches anything) with just 1s
+ // Optimised Peq building avoiding branching.
+ for (int symbol = 0; symbol < alphabetLength; symbol++) {
+ for (int b = 0; b < maxNumBlocks; b++) {
+ Word PeqW = 0;
+ for (int r = (b+1) * WORD_SIZE - 1; r >= b * WORD_SIZE; r--) {
+ PeqW = (PeqW<<1)
+ + (r >= queryLength
+ || equalityDefinition_areEqual(equalityDefinition,
+ query[r], symbol));
+ }
+ Peq[symbol * maxNumBlocks + b] = PeqW;
+ }
+ }
+ {
+ int symbol = alphabetLength;
+ for (int b = 0; b < maxNumBlocks; b++) {
+ // Last symbol is wildcard, so it is all 1s
+ Peq[symbol * maxNumBlocks + b] = (Word)-1;
+ }
+ }
+
+ return Peq;
+}
+
+
+/**
+ * Returns new sequence that is reverse of given sequence.
+ * Free returned array with free()
+ */
+static inline unsigned char* createReverseCopy(const unsigned char* const seq, const int length) {
+ unsigned char* rSeq = malloc(length);
+ for (int i = 0; i < length; i++) {
+ rSeq[i] = seq[length - i - 1];
+ }
+ return rSeq;
+}
+
+/**
+ * Corresponds to Advance_Block function from Myers.
+ * Calculates one word(block), which is part of a column.
+ * Highest bit of word (one most to the left) is most bottom cell of block from column.
+ * Pv[i] and Mv[i] define vin of cell[i]: vin = cell[i] - cell[i-1].
+ * @param [in] Pv Bitset, Pv[i] == 1 if vin is +1, otherwise Pv[i] == 0.
+ * @param [in] Mv Bitset, Mv[i] == 1 if vin is -1, otherwise Mv[i] == 0.
+ * @param [in] Eq Bitset, Eq[i] == 1 if match, 0 if mismatch.
+ * @param [in] hin Will be +1, 0 or -1.
+ * @param [out] PvOut Bitset, PvOut[i] == 1 if vout is +1, otherwise PvOut[i] == 0.
+ * @param [out] MvOut Bitset, MvOut[i] == 1 if vout is -1, otherwise MvOut[i] == 0.
+ * @param [out] hout Will be +1, 0 or -1.
+ */
+static inline int calculateBlock(Word Pv, Word Mv, Word Eq, const int hin,
+ Word *PvOut, Word *MvOut) {
+ // hin can be 1, -1 or 0.
+ // 1 -> 00...01
+ // 0 -> 00...00
+ // -1 -> 11...11 (2-complement)
+
+ Word hinIsNeg = (Word)(hin >> 2) & WORD_1; // 00...001 if hin is -1, 00...000 if 0 or 1
+
+ Word Xv = Eq | Mv;
+ // This is instruction below written using 'if': if (hin < 0) Eq |= (Word)1;
+ Eq |= hinIsNeg;
+ Word Xh = (((Eq & Pv) + Pv) ^ Pv) | Eq;
+
+ Word Ph = Mv | ~(Xh | Pv);
+ Word Mh = Pv & Xh;
+
+ int hout = 0;
+ // This is instruction below written using 'if': if (Ph & HIGH_BIT_MASK) hout = 1;
+ hout = (Ph & HIGH_BIT_MASK) >> (WORD_SIZE - 1);
+ // This is instruction below written using 'if': if (Mh & HIGH_BIT_MASK) hout = -1;
+ hout -= (Mh & HIGH_BIT_MASK) >> (WORD_SIZE - 1);
+
+ Ph <<= 1;
+ Mh <<= 1;
+
+ // This is instruction below written using 'if': if (hin < 0) Mh |= (Word)1;
+ Mh |= hinIsNeg;
+ // This is instruction below written using 'if': if (hin > 0) Ph |= (Word)1;
+ Ph |= (Word)((hin + 1) >> 1);
+
+ *PvOut = Mh | ~(Xv | Ph);
+ *MvOut = Ph & Xv;
+
+ return hout;
+}
+
+/**
+ * Does ceiling division x / y.
+ * Note: x and y must be non-negative and x + y must not overflow.
+ */
+static inline int ceilDiv(const int x, const int y) {
+ return x % y ? x / y + 1 : x / y;
+}
+
+static inline int min(const int x, const int y) {
+ return x < y ? x : y;
+}
+
+
+/**
+ * @param [in] block
+ * @return Values of cells in block, starting with bottom cell in block.
+ */
+static inline int *getBlockCellValues(const Block block) {
+ int *scores = malloc(WORD_SIZE * sizeof(*scores));
+ int score = block.score;
+ Word mask = HIGH_BIT_MASK;
+ for (int i = 0; i < WORD_SIZE - 1; i++) {
+ scores[i] = score;
+ if (block.P & mask) score--;
+ if (block.M & mask) score++;
+ mask >>= 1;
+ }
+ scores[WORD_SIZE - 1] = score;
+ return scores;
+}
+
+/**
+ * @param [in] block
+ * @param [in] k
+ * @return True if all cells in block have value larger than k, otherwise false.
+ */
+static inline bool allBlockCellsLarger(const Block block, const int k) {
+ int *scores = getBlockCellValues(block);
+ for (int i = 0; i < WORD_SIZE; i++) {
+ if (scores[i] <= k) {
+ free(scores);
+ return false;
+ }
+ }
+
+ free(scores);
+ return true;
+}
+
+
+/**
+ * Uses Myers' bit-vector algorithm to find edit distance for one of semi-global alignment methods.
+ * @param [in] Peq Query profile.
+ * @param [in] W Size of padding in last block.
+ * TODO: Calculate this directly from query, instead of passing it.
+ * @param [in] maxNumBlocks Number of blocks needed to cover the whole query.
+ * TODO: Calculate this directly from query, instead of passing it.
+ * @param [in] queryLength
+ * @param [in] target
+ * @param [in] targetLength
+ * @param [in] k
+ * @param [in] mode EDLIB_MODE_HW or EDLIB_MODE_SHW
+ * @param [out] bestScore_ Edit distance.
+ * @param [out] positions_ Array of 0-indexed positions in target at which best score was found.
+ Make sure to free this array with free().
+ * @param [out] numPositions_ Number of positions in the positions_ array.
+ * @return Status.
+ */
+static int myersCalcEditDistanceSemiGlobal(
+ const Word* const Peq, const int W, const int maxNumBlocks,
+ const int queryLength,
+ const unsigned char* const target, const int targetLength,
+ int k, const EdlibAlignMode mode,
+ int* const bestScore_, int** const positions_, int* const numPositions_) {
+ *positions_ = NULL;
+ *numPositions_ = 0;
+
+ // firstBlock is 0-based index of first block in Ukkonen band.
+ // lastBlock is 0-based index of last block in Ukkonen band.
+ int firstBlock = 0;
+ int lastBlock = min(ceilDiv(k + 1, WORD_SIZE), maxNumBlocks) - 1; // y in Myers
+ Block *bl; // Current block
+
+ Block* blocks = malloc(maxNumBlocks * sizeof(*blocks));
+
+ // For HW, solution will never be larger then queryLength.
+ if (mode == EDLIB_MODE_HW) {
+ k = min(queryLength, k);
+ }
+
+ // Each STRONG_REDUCE_NUM column is reduced in more expensive way.
+ // This gives speed up of about 2 times for small k.
+ const int STRONG_REDUCE_NUM = 2048;
+
+ // Initialize P, M and score
+ bl = blocks;
+ for (int b = 0; b <= lastBlock; b++) {
+ bl->score = (b + 1) * WORD_SIZE;
+ bl->P = (Word)(-1); // All 1s
+ bl->M = (Word)(0);
+ bl++;
+ }
+
+ int bestScore = -1;
+#define MAX_POS 100 // maximum number of positions returned.
+ int positions[MAX_POS];
+ int npositions = 0;
+ const int startHout = mode == EDLIB_MODE_HW ? 0 : 1; // If 0 then gap before query is not penalized;
+ const unsigned char* targetChar = target;
+ for (int c = 0; c < targetLength; c++) { // for each column
+ const Word* Peq_c = Peq + (*targetChar) * maxNumBlocks;
+
+ //----------------------- Calculate column -------------------------//
+ int hout = startHout;
+ bl = blocks + firstBlock;
+ Peq_c += firstBlock;
+ for (int b = firstBlock; b <= lastBlock; b++) {
+ hout = calculateBlock(bl->P, bl->M, *Peq_c, hout, &bl->P, &bl->M);
+ bl->score += hout;
+ bl++; Peq_c++;
+ }
+ bl--; Peq_c--;
+ //------------------------------------------------------------------//
+
+ //---------- Adjust number of blocks according to Ukkonen ----------//
+ if ((lastBlock < maxNumBlocks - 1) && (bl->score - hout <= k) // bl is pointing to last block
+ && ((*(Peq_c + 1) & WORD_1) || hout < 0)) { // Peq_c is pointing to last block
+ // If score of left block is not too big, calculate one more block
+ lastBlock++; bl++; Peq_c++;
+ bl->P = (Word)(-1); // All 1s
+ bl->M = (Word)(0);
+ bl->score = (bl - 1)->score - hout + WORD_SIZE + calculateBlock(bl->P, bl->M, *Peq_c, hout, &bl->P, &bl->M);
+ } else {
+ while (lastBlock >= firstBlock && bl->score >= k + WORD_SIZE) {
+ lastBlock--; bl--; Peq_c--;
+ }
+ }
+
+ // Every some columns, do some expensive but also more efficient block reducing.
+ // This is important!
+ //
+ // Reduce the band by decreasing last block if possible.
+ if (c % STRONG_REDUCE_NUM == 0) {
+ while (lastBlock >= 0 && lastBlock >= firstBlock && allBlockCellsLarger(*bl, k)) {
+ lastBlock--; bl--; Peq_c--;
+ }
+ }
+ // For HW, even if all cells are > k, there still may be solution in next
+ // column because starting conditions at upper boundary are 0.
+ // That means that first block is always candidate for solution,
+ // and we can never end calculation before last column.
+ if (mode == EDLIB_MODE_HW && lastBlock == -1) {
+ lastBlock++; bl++; Peq_c++;
+ }
+
+ // Reduce band by increasing first block if possible. Not applicable to HW.
+ if (mode != EDLIB_MODE_HW) {
+ while (firstBlock <= lastBlock && blocks[firstBlock].score >= k + WORD_SIZE) {
+ firstBlock++;
+ }
+ if (c % STRONG_REDUCE_NUM == 0) { // Do strong reduction every some blocks
+ while (firstBlock <= lastBlock && allBlockCellsLarger(blocks[firstBlock], k)) {
+ firstBlock++;
+ }
+ }
+ }
+
+ // If band stops to exist finish
+ if (lastBlock < firstBlock) {
+ *bestScore_ = bestScore;
+ if (bestScore != -1) {
+ *positions_ = malloc(npositions * sizeof(int));
+ *numPositions_ = npositions;
+ memcpy(*positions_, positions, npositions * sizeof(int));
+ }
+ free(blocks);
+ return EDLIB_STATUS_OK;
+ }
+ //------------------------------------------------------------------//
+
+ //------------------------- Update best score ----------------------//
+ if (lastBlock == maxNumBlocks - 1) {
+ int colScore = bl->score;
+ if (colScore <= k) { // Scores > k dont have correct values (so we cannot use them), but are certainly > k.
+ // NOTE: Score that I find in column c is actually score from column c-W
+ if (bestScore == -1 || colScore <= bestScore) {
+ if (colScore != bestScore) {
+ npositions = 0;
+ bestScore = colScore;
+ // Change k so we will look only for equal or better
+ // scores then the best found so far.
+ k = bestScore;
+ }
+ if (npositions < MAX_POS)
+ positions[npositions++] = c - W;
+ }
+ }
+ }
+ //------------------------------------------------------------------//
+
+ targetChar++;
+ }
+
+
+ // Obtain results for last W columns from last column.
+ if (lastBlock == maxNumBlocks - 1) {
+ int *blockScores = getBlockCellValues(*bl);
+ for (int i = 0; i < W; i++) {
+ int colScore = blockScores[i + 1];
+ if (colScore <= k && (bestScore == -1 || colScore <= bestScore)) {
+ if (colScore != bestScore) {
+ npositions = 0;
+ k = bestScore = colScore;
+ }
+ if (npositions < MAX_POS)
+ positions[npositions++] = targetLength - W + i;
+ }
+ }
+ free(blockScores);
+ }
+
+ *bestScore_ = bestScore;
+ if (bestScore != -1) {
+ *positions_ = malloc(npositions * sizeof(int));
+ *numPositions_ = npositions;
+ memcpy(*positions_, positions, npositions * sizeof(int));
+ }
+
+ free(blocks);
+ return EDLIB_STATUS_OK;
+}
+
+
+/**
+ * Takes char query and char target, recognizes alphabet and transforms them into unsigned char sequences
+ * where elements in sequences are not any more letters of alphabet, but their index in alphabet.
+ * Most of internal edlib functions expect such transformed sequences.
+ * This function will allocate queryTransformed and targetTransformed, so make sure to free them when done.
+ * Example:
+ * Original sequences: "ACT" and "CGT".
+ * Alphabet would be recognized as "ACTG". Alphabet length = 4.
+ * Transformed sequences: [0, 1, 2] and [1, 3, 2].
+ * @param [in] queryOriginal
+ * @param [in] queryLength
+ * @param [in] targetOriginal
+ * @param [in] targetLength
+ * @param [out] queryTransformed It will contain values in range [0, alphabet length - 1].
+ * @param [out] targetTransformed It will contain values in range [0, alphabet length - 1].
+ * @return Alphabet as a string of unique characters, where index of each character is its value in transformed
+ * sequences.
+ */
+static char *transformSequences(const char* const queryOriginal, const int queryLength,
+ const char* const targetOriginal, const int targetLength,
+ unsigned char** const queryTransformed,
+ unsigned char** const targetTransformed,
+ int *alphabet_size) {
+ // Alphabet is constructed from letters that are present in sequences.
+ // Each letter is assigned an ordinal number, starting from 0 up to alphabetLength - 1,
+ // and new query and target are created in which letters are replaced with their ordinal numbers.
+ // This query and target are used in all the calculations later.
+ *queryTransformed = malloc(sizeof(unsigned char) * queryLength);
+ *targetTransformed = malloc(sizeof(unsigned char) * targetLength);
+
+ char *alphabet = malloc(MAX_UCHAR+1), *alphabet_cp = alphabet;
+
+ // Alphabet information, it is constructed on fly while transforming sequences.
+ // letterIdx[c] is index of letter c in alphabet.
+ unsigned char letterIdx[MAX_UCHAR + 1];
+ bool inAlphabet[MAX_UCHAR + 1]; // inAlphabet[c] is true if c is in alphabet
+ for (int i = 0; i < MAX_UCHAR + 1; i++) inAlphabet[i] = false;
+
+ for (int i = 0; i < queryLength; i++) {
+ unsigned char c = queryOriginal[i];
+ if (!inAlphabet[c]) {
+ inAlphabet[c] = true;
+ letterIdx[c] = alphabet_cp - alphabet;
+ *alphabet_cp++ = queryOriginal[i];
+ }
+ (*queryTransformed)[i] = letterIdx[c];
+ }
+ for (int i = 0; i < targetLength; i++) {
+ unsigned char c = targetOriginal[i];
+ if (!inAlphabet[c]) {
+ inAlphabet[c] = true;
+ letterIdx[c] = alphabet_cp - alphabet;
+ *alphabet_cp++ = targetOriginal[i];
+ }
+ (*targetTransformed)[i] = letterIdx[c];
+ }
+
+ *alphabet_size = alphabet_cp - alphabet;
+ return alphabet;
+}
+
+
+EdlibAlignConfig edlibNewAlignConfig(int k, EdlibAlignMode mode, EdlibAlignTask task,
+ const EdlibEqualityPair* additionalEqualities,
+ int additionalEqualitiesLength) {
+ EdlibAlignConfig config;
+ config.k = k;
+ config.mode = mode;
+ config.task = task;
+ config.additionalEqualities = additionalEqualities;
+ config.additionalEqualitiesLength = additionalEqualitiesLength;
+ return config;
+}
+
+EdlibAlignConfig edlibDefaultAlignConfig(void) {
+ return edlibNewAlignConfig(-1, EDLIB_MODE_NW, EDLIB_TASK_DISTANCE, NULL, 0);
+}
+
+void edlibFreeAlignResult(EdlibAlignResult result) {
+ if (result.endLocations) free(result.endLocations);
+ if (result.startLocations) free(result.startLocations);
+ if (result.alignment) free(result.alignment);
+}
--- /dev/null
+#include "bcftools.pysam.h"
+
+/*
+ * A cut down C translated of the C++ edlib.cpp file.
+ * Taken from edlib v0.1.0-166-g931be2b
+ */
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdbool.h>
+
+#include "edlib.h"
+
+typedef uint64_t Word;
+static const int WORD_SIZE = 64; // Size of Word in bits
+static const Word WORD_1 = (Word)1;
+static const Word HIGH_BIT_MASK = 1LL << 63; // 100..00
+//#define MAX_UCHAR 255
+#define MAX_UCHAR 7 // better cache usage for our data
+
+#ifndef MAX
+#define MAX(a,b) ((a)>(b)?(a):(b))
+#endif
+
+typedef struct Block {
+ Word P; // Pvin
+ Word M; // Mvin
+ int score; // score of last cell in block;
+} Block;
+
+
+/**
+ * Defines equality relation on alphabet characters.
+ * By default each character is always equal only to itself, but you can also provide additional equalities.
+ */
+typedef struct EqualityDefinition {
+ bool matrix[MAX_UCHAR + 1][MAX_UCHAR + 1];
+} EqualityDefinition;
+
+static EqualityDefinition *
+CreateEqualityDefinition(const char *alphabet, int alphabet_size,
+ const EdlibEqualityPair* additionalEqualities,
+ const int additionalEqualitiesLength) {
+ EqualityDefinition *ed = malloc(sizeof(*ed));
+
+ for (size_t i = 0; i < alphabet_size; i++) {
+ for (size_t j = 0; j < alphabet_size; j++) {
+ ed->matrix[i][j] = (i == j);
+ }
+ }
+ if (additionalEqualities != NULL) {
+ for (int i = 0; i < additionalEqualitiesLength; i++) {
+ const char *firstTransformed = strchr(alphabet, additionalEqualities[i].first);
+ const char *secondTransformed = strchr(alphabet, additionalEqualities[i].second);
+ if (firstTransformed && alphabet_size) {
+ ed->matrix[firstTransformed - alphabet][secondTransformed - alphabet] =
+ ed->matrix[secondTransformed - alphabet][firstTransformed - alphabet]
+ = true;
+ }
+ }
+ }
+
+ return ed;
+}
+
+/**
+ * @param a Element from transformed sequence.
+ * @param b Element from transformed sequence.
+ * @return True if a and b are defined as equal, false otherwise.
+ */
+static inline const /* attribute pure or const? */
+bool equalityDefinition_areEqual(const EqualityDefinition *ed, unsigned char a, unsigned char b) {
+ return ed->matrix[a][b];
+}
+
+static int myersCalcEditDistanceSemiGlobal(const Word* Peq, int W, int maxNumBlocks,
+ int queryLength,
+ const unsigned char* target, int targetLength,
+ int k, EdlibAlignMode mode,
+ int* bestScore_, int** positions_, int* numPositions_);
+
+static char *transformSequences(const char* queryOriginal, int queryLength,
+ const char* targetOriginal, int targetLength,
+ unsigned char** queryTransformed,
+ unsigned char** targetTransformed,
+ int *alphabet_size);
+
+static inline int ceilDiv(int x, int y);
+
+static inline unsigned char* createReverseCopy(const unsigned char* seq, int length);
+
+static inline Word* buildPeq(const int alphabetLength,
+ const unsigned char* query,
+ const int queryLength,
+ const EqualityDefinition* equalityDefinition);
+
+
+/**
+ * Main edlib method.
+ */
+EdlibAlignResult edlibAlign(const char* const queryOriginal, const int queryLength,
+ const char* const targetOriginal, const int targetLength,
+ const EdlibAlignConfig config) {
+ EdlibAlignResult result;
+ result.status = EDLIB_STATUS_OK;
+ result.editDistance = -1;
+ result.endLocations = result.startLocations = NULL;
+ result.numLocations = 0;
+ result.alignment = NULL;
+ result.alignmentLength = 0;
+ result.alphabetLength = 0;
+
+ /*------------ TRANSFORM SEQUENCES AND RECOGNIZE ALPHABET -----------*/
+ unsigned char* query, * target;
+ int alphabet_size;
+ char *alphabet = transformSequences(queryOriginal, queryLength, targetOriginal, targetLength,
+ &query, &target, &alphabet_size);
+ result.alphabetLength = alphabet_size;
+ /*-------------------------------------------------------*/
+
+ // Handle special situation when at least one of the sequences has length 0.
+ if (queryLength == 0 || targetLength == 0) {
+ if (config.mode == EDLIB_MODE_NW) {
+ result.editDistance = MAX(queryLength, targetLength);
+ result.endLocations = malloc(sizeof(int) * 1);
+ result.endLocations[0] = targetLength - 1;
+ result.numLocations = 1;
+ } else if (config.mode == EDLIB_MODE_SHW || config.mode == EDLIB_MODE_HW) {
+ result.editDistance = queryLength;
+ result.endLocations = malloc(sizeof(int) * 1);
+ result.endLocations[0] = -1;
+ result.numLocations = 1;
+ } else {
+ result.status = EDLIB_STATUS_ERROR;
+ }
+
+ free(query);
+ free(target);
+ free(alphabet);
+ return result;
+ }
+
+ /*--------------------- INITIALIZATION ------------------*/
+ int maxNumBlocks = ceilDiv(queryLength, WORD_SIZE); // bmax in Myers
+ int W = maxNumBlocks * WORD_SIZE - queryLength; // number of redundant cells in last level blocks
+ EqualityDefinition *equalityDefinition =
+ CreateEqualityDefinition(alphabet, alphabet_size, config.additionalEqualities, config.additionalEqualitiesLength);
+ Word* Peq = buildPeq(alphabet_size, query, queryLength, equalityDefinition);
+ /*-------------------------------------------------------*/
+
+ /*------------------ MAIN CALCULATION -------------------*/
+ // TODO: Store alignment data only after k is determined? That could make things faster.
+// int positionNW; // Used only when mode is NW.
+// AlignmentData* alignData = NULL;
+ bool dynamicK = false;
+ int k = config.k;
+ if (k < 0) { // If valid k is not given, auto-adjust k until solution is found.
+ dynamicK = true;
+ k = WORD_SIZE; // Gives better results than smaller k.
+ }
+
+ do {
+ if (config.mode == EDLIB_MODE_HW || config.mode == EDLIB_MODE_SHW) {
+ myersCalcEditDistanceSemiGlobal(Peq, W, maxNumBlocks,
+ queryLength, target, targetLength,
+ k, config.mode, &(result.editDistance),
+ &(result.endLocations), &(result.numLocations));
+ } else { // mode == EDLIB_MODE_NW
+// myersCalcEditDistanceNW(Peq, W, maxNumBlocks,
+// queryLength, target, targetLength,
+// k, &(result.editDistance), &positionNW,
+// false, &alignData, -1);
+ }
+ k *= 2;
+ } while(dynamicK && result.editDistance == -1);
+
+ if (result.editDistance >= 0) { // If there is solution.
+ // If NW mode, set end location explicitly.
+ if (config.mode == EDLIB_MODE_NW) {
+ result.endLocations = malloc(sizeof(int) * 1);
+ result.endLocations[0] = targetLength - 1;
+ result.numLocations = 1;
+ }
+
+ // Find starting locations.
+ if (config.task == EDLIB_TASK_LOC || config.task == EDLIB_TASK_PATH) {
+ result.startLocations = malloc(result.numLocations * sizeof(int));
+ if (config.mode == EDLIB_MODE_HW) { // If HW, I need to calculate start locations.
+ const unsigned char* rTarget = createReverseCopy(target, targetLength);
+ const unsigned char* rQuery = createReverseCopy(query, queryLength);
+ // Peq for reversed query.
+ Word* rPeq = buildPeq(alphabet_size, rQuery, queryLength, equalityDefinition);
+ for (int i = 0; i < result.numLocations; i++) {
+ int endLocation = result.endLocations[i];
+ if (endLocation == -1) {
+ // NOTE: Sometimes one of optimal solutions is that query starts before target, like this:
+ // AAGG <- target
+ // CCTT <- query
+ // It will never be only optimal solution and it does not happen often, however it is
+ // possible and in that case end location will be -1. What should we do with that?
+ // Should we just skip reporting such end location, although it is a solution?
+ // If we do report it, what is the start location? -4? -1? Nothing?
+ // TODO: Figure this out. This has to do in general with how we think about start
+ // and end locations.
+ // Also, we have alignment later relying on this locations to limit the space of it's
+ // search -> how can it do it right if these locations are negative or incorrect?
+ result.startLocations[i] = 0; // I put 0 for now, but it does not make much sense.
+ } else {
+ int bestScoreSHW, numPositionsSHW;
+ int* positionsSHW;
+ myersCalcEditDistanceSemiGlobal(
+ rPeq, W, maxNumBlocks,
+ queryLength, rTarget + targetLength - endLocation - 1, endLocation + 1,
+ result.editDistance, EDLIB_MODE_SHW,
+ &bestScoreSHW, &positionsSHW, &numPositionsSHW);
+ // Taking last location as start ensures that alignment will not start with insertions
+ // if it can start with mismatches instead.
+ result.startLocations[i] = endLocation - positionsSHW[numPositionsSHW - 1];
+ free(positionsSHW);
+ }
+ }
+ free((void *)rTarget);
+ free((void *)rQuery);
+ free(rPeq);
+ } else { // If mode is SHW or NW
+ for (int i = 0; i < result.numLocations; i++) {
+ result.startLocations[i] = 0;
+ }
+ }
+ }
+ }
+ /*-------------------------------------------------------*/
+
+ //--- Free memory ---//
+ free(Peq);
+ free(query);
+ free(target);
+ free(alphabet);
+ free(equalityDefinition);
+// DestroyAlignmentData(alignData);
+ //-------------------//
+
+ return result;
+}
+
+/**
+ * Build Peq table for given query and alphabet.
+ * Peq is table of dimensions alphabetLength+1 x maxNumBlocks.
+ * Bit i of Peq[s * maxNumBlocks + b] is 1 if i-th symbol from block b of query equals symbol s, otherwise it is 0.
+ * NOTICE: free returned array with free()!
+ */
+static inline Word* buildPeq(const int alphabetLength,
+ const unsigned char* const query,
+ const int queryLength,
+ const EqualityDefinition* equalityDefinition) {
+ int maxNumBlocks = ceilDiv(queryLength, WORD_SIZE);
+ // table of dimensions alphabetLength+1 x maxNumBlocks. Last symbol is wildcard.
+ Word* Peq = malloc((alphabetLength + 1) * maxNumBlocks * sizeof(*Peq));
+
+ // Build Peq (1 is match, 0 is mismatch). NOTE: last column is wildcard(symbol that matches anything) with just 1s
+ // Optimised Peq building avoiding branching.
+ for (int symbol = 0; symbol < alphabetLength; symbol++) {
+ for (int b = 0; b < maxNumBlocks; b++) {
+ Word PeqW = 0;
+ for (int r = (b+1) * WORD_SIZE - 1; r >= b * WORD_SIZE; r--) {
+ PeqW = (PeqW<<1)
+ + (r >= queryLength
+ || equalityDefinition_areEqual(equalityDefinition,
+ query[r], symbol));
+ }
+ Peq[symbol * maxNumBlocks + b] = PeqW;
+ }
+ }
+ {
+ int symbol = alphabetLength;
+ for (int b = 0; b < maxNumBlocks; b++) {
+ // Last symbol is wildcard, so it is all 1s
+ Peq[symbol * maxNumBlocks + b] = (Word)-1;
+ }
+ }
+
+ return Peq;
+}
+
+
+/**
+ * Returns new sequence that is reverse of given sequence.
+ * Free returned array with free()
+ */
+static inline unsigned char* createReverseCopy(const unsigned char* const seq, const int length) {
+ unsigned char* rSeq = malloc(length);
+ for (int i = 0; i < length; i++) {
+ rSeq[i] = seq[length - i - 1];
+ }
+ return rSeq;
+}
+
+/**
+ * Corresponds to Advance_Block function from Myers.
+ * Calculates one word(block), which is part of a column.
+ * Highest bit of word (one most to the left) is most bottom cell of block from column.
+ * Pv[i] and Mv[i] define vin of cell[i]: vin = cell[i] - cell[i-1].
+ * @param [in] Pv Bitset, Pv[i] == 1 if vin is +1, otherwise Pv[i] == 0.
+ * @param [in] Mv Bitset, Mv[i] == 1 if vin is -1, otherwise Mv[i] == 0.
+ * @param [in] Eq Bitset, Eq[i] == 1 if match, 0 if mismatch.
+ * @param [in] hin Will be +1, 0 or -1.
+ * @param [out] PvOut Bitset, PvOut[i] == 1 if vout is +1, otherwise PvOut[i] == 0.
+ * @param [out] MvOut Bitset, MvOut[i] == 1 if vout is -1, otherwise MvOut[i] == 0.
+ * @param [out] hout Will be +1, 0 or -1.
+ */
+static inline int calculateBlock(Word Pv, Word Mv, Word Eq, const int hin,
+ Word *PvOut, Word *MvOut) {
+ // hin can be 1, -1 or 0.
+ // 1 -> 00...01
+ // 0 -> 00...00
+ // -1 -> 11...11 (2-complement)
+
+ Word hinIsNeg = (Word)(hin >> 2) & WORD_1; // 00...001 if hin is -1, 00...000 if 0 or 1
+
+ Word Xv = Eq | Mv;
+ // This is instruction below written using 'if': if (hin < 0) Eq |= (Word)1;
+ Eq |= hinIsNeg;
+ Word Xh = (((Eq & Pv) + Pv) ^ Pv) | Eq;
+
+ Word Ph = Mv | ~(Xh | Pv);
+ Word Mh = Pv & Xh;
+
+ int hout = 0;
+ // This is instruction below written using 'if': if (Ph & HIGH_BIT_MASK) hout = 1;
+ hout = (Ph & HIGH_BIT_MASK) >> (WORD_SIZE - 1);
+ // This is instruction below written using 'if': if (Mh & HIGH_BIT_MASK) hout = -1;
+ hout -= (Mh & HIGH_BIT_MASK) >> (WORD_SIZE - 1);
+
+ Ph <<= 1;
+ Mh <<= 1;
+
+ // This is instruction below written using 'if': if (hin < 0) Mh |= (Word)1;
+ Mh |= hinIsNeg;
+ // This is instruction below written using 'if': if (hin > 0) Ph |= (Word)1;
+ Ph |= (Word)((hin + 1) >> 1);
+
+ *PvOut = Mh | ~(Xv | Ph);
+ *MvOut = Ph & Xv;
+
+ return hout;
+}
+
+/**
+ * Does ceiling division x / y.
+ * Note: x and y must be non-negative and x + y must not overflow.
+ */
+static inline int ceilDiv(const int x, const int y) {
+ return x % y ? x / y + 1 : x / y;
+}
+
+static inline int min(const int x, const int y) {
+ return x < y ? x : y;
+}
+
+
+/**
+ * @param [in] block
+ * @return Values of cells in block, starting with bottom cell in block.
+ */
+static inline int *getBlockCellValues(const Block block) {
+ int *scores = malloc(WORD_SIZE * sizeof(*scores));
+ int score = block.score;
+ Word mask = HIGH_BIT_MASK;
+ for (int i = 0; i < WORD_SIZE - 1; i++) {
+ scores[i] = score;
+ if (block.P & mask) score--;
+ if (block.M & mask) score++;
+ mask >>= 1;
+ }
+ scores[WORD_SIZE - 1] = score;
+ return scores;
+}
+
+/**
+ * @param [in] block
+ * @param [in] k
+ * @return True if all cells in block have value larger than k, otherwise false.
+ */
+static inline bool allBlockCellsLarger(const Block block, const int k) {
+ int *scores = getBlockCellValues(block);
+ for (int i = 0; i < WORD_SIZE; i++) {
+ if (scores[i] <= k) {
+ free(scores);
+ return false;
+ }
+ }
+
+ free(scores);
+ return true;
+}
+
+
+/**
+ * Uses Myers' bit-vector algorithm to find edit distance for one of semi-global alignment methods.
+ * @param [in] Peq Query profile.
+ * @param [in] W Size of padding in last block.
+ * TODO: Calculate this directly from query, instead of passing it.
+ * @param [in] maxNumBlocks Number of blocks needed to cover the whole query.
+ * TODO: Calculate this directly from query, instead of passing it.
+ * @param [in] queryLength
+ * @param [in] target
+ * @param [in] targetLength
+ * @param [in] k
+ * @param [in] mode EDLIB_MODE_HW or EDLIB_MODE_SHW
+ * @param [out] bestScore_ Edit distance.
+ * @param [out] positions_ Array of 0-indexed positions in target at which best score was found.
+ Make sure to free this array with free().
+ * @param [out] numPositions_ Number of positions in the positions_ array.
+ * @return Status.
+ */
+static int myersCalcEditDistanceSemiGlobal(
+ const Word* const Peq, const int W, const int maxNumBlocks,
+ const int queryLength,
+ const unsigned char* const target, const int targetLength,
+ int k, const EdlibAlignMode mode,
+ int* const bestScore_, int** const positions_, int* const numPositions_) {
+ *positions_ = NULL;
+ *numPositions_ = 0;
+
+ // firstBlock is 0-based index of first block in Ukkonen band.
+ // lastBlock is 0-based index of last block in Ukkonen band.
+ int firstBlock = 0;
+ int lastBlock = min(ceilDiv(k + 1, WORD_SIZE), maxNumBlocks) - 1; // y in Myers
+ Block *bl; // Current block
+
+ Block* blocks = malloc(maxNumBlocks * sizeof(*blocks));
+
+ // For HW, solution will never be larger then queryLength.
+ if (mode == EDLIB_MODE_HW) {
+ k = min(queryLength, k);
+ }
+
+ // Each STRONG_REDUCE_NUM column is reduced in more expensive way.
+ // This gives speed up of about 2 times for small k.
+ const int STRONG_REDUCE_NUM = 2048;
+
+ // Initialize P, M and score
+ bl = blocks;
+ for (int b = 0; b <= lastBlock; b++) {
+ bl->score = (b + 1) * WORD_SIZE;
+ bl->P = (Word)(-1); // All 1s
+ bl->M = (Word)(0);
+ bl++;
+ }
+
+ int bestScore = -1;
+#define MAX_POS 100 // maximum number of positions returned.
+ int positions[MAX_POS];
+ int npositions = 0;
+ const int startHout = mode == EDLIB_MODE_HW ? 0 : 1; // If 0 then gap before query is not penalized;
+ const unsigned char* targetChar = target;
+ for (int c = 0; c < targetLength; c++) { // for each column
+ const Word* Peq_c = Peq + (*targetChar) * maxNumBlocks;
+
+ //----------------------- Calculate column -------------------------//
+ int hout = startHout;
+ bl = blocks + firstBlock;
+ Peq_c += firstBlock;
+ for (int b = firstBlock; b <= lastBlock; b++) {
+ hout = calculateBlock(bl->P, bl->M, *Peq_c, hout, &bl->P, &bl->M);
+ bl->score += hout;
+ bl++; Peq_c++;
+ }
+ bl--; Peq_c--;
+ //------------------------------------------------------------------//
+
+ //---------- Adjust number of blocks according to Ukkonen ----------//
+ if ((lastBlock < maxNumBlocks - 1) && (bl->score - hout <= k) // bl is pointing to last block
+ && ((*(Peq_c + 1) & WORD_1) || hout < 0)) { // Peq_c is pointing to last block
+ // If score of left block is not too big, calculate one more block
+ lastBlock++; bl++; Peq_c++;
+ bl->P = (Word)(-1); // All 1s
+ bl->M = (Word)(0);
+ bl->score = (bl - 1)->score - hout + WORD_SIZE + calculateBlock(bl->P, bl->M, *Peq_c, hout, &bl->P, &bl->M);
+ } else {
+ while (lastBlock >= firstBlock && bl->score >= k + WORD_SIZE) {
+ lastBlock--; bl--; Peq_c--;
+ }
+ }
+
+ // Every some columns, do some expensive but also more efficient block reducing.
+ // This is important!
+ //
+ // Reduce the band by decreasing last block if possible.
+ if (c % STRONG_REDUCE_NUM == 0) {
+ while (lastBlock >= 0 && lastBlock >= firstBlock && allBlockCellsLarger(*bl, k)) {
+ lastBlock--; bl--; Peq_c--;
+ }
+ }
+ // For HW, even if all cells are > k, there still may be solution in next
+ // column because starting conditions at upper boundary are 0.
+ // That means that first block is always candidate for solution,
+ // and we can never end calculation before last column.
+ if (mode == EDLIB_MODE_HW && lastBlock == -1) {
+ lastBlock++; bl++; Peq_c++;
+ }
+
+ // Reduce band by increasing first block if possible. Not applicable to HW.
+ if (mode != EDLIB_MODE_HW) {
+ while (firstBlock <= lastBlock && blocks[firstBlock].score >= k + WORD_SIZE) {
+ firstBlock++;
+ }
+ if (c % STRONG_REDUCE_NUM == 0) { // Do strong reduction every some blocks
+ while (firstBlock <= lastBlock && allBlockCellsLarger(blocks[firstBlock], k)) {
+ firstBlock++;
+ }
+ }
+ }
+
+ // If band stops to exist finish
+ if (lastBlock < firstBlock) {
+ *bestScore_ = bestScore;
+ if (bestScore != -1) {
+ *positions_ = malloc(npositions * sizeof(int));
+ *numPositions_ = npositions;
+ memcpy(*positions_, positions, npositions * sizeof(int));
+ }
+ free(blocks);
+ return EDLIB_STATUS_OK;
+ }
+ //------------------------------------------------------------------//
+
+ //------------------------- Update best score ----------------------//
+ if (lastBlock == maxNumBlocks - 1) {
+ int colScore = bl->score;
+ if (colScore <= k) { // Scores > k dont have correct values (so we cannot use them), but are certainly > k.
+ // NOTE: Score that I find in column c is actually score from column c-W
+ if (bestScore == -1 || colScore <= bestScore) {
+ if (colScore != bestScore) {
+ npositions = 0;
+ bestScore = colScore;
+ // Change k so we will look only for equal or better
+ // scores then the best found so far.
+ k = bestScore;
+ }
+ if (npositions < MAX_POS)
+ positions[npositions++] = c - W;
+ }
+ }
+ }
+ //------------------------------------------------------------------//
+
+ targetChar++;
+ }
+
+
+ // Obtain results for last W columns from last column.
+ if (lastBlock == maxNumBlocks - 1) {
+ int *blockScores = getBlockCellValues(*bl);
+ for (int i = 0; i < W; i++) {
+ int colScore = blockScores[i + 1];
+ if (colScore <= k && (bestScore == -1 || colScore <= bestScore)) {
+ if (colScore != bestScore) {
+ npositions = 0;
+ k = bestScore = colScore;
+ }
+ if (npositions < MAX_POS)
+ positions[npositions++] = targetLength - W + i;
+ }
+ }
+ free(blockScores);
+ }
+
+ *bestScore_ = bestScore;
+ if (bestScore != -1) {
+ *positions_ = malloc(npositions * sizeof(int));
+ *numPositions_ = npositions;
+ memcpy(*positions_, positions, npositions * sizeof(int));
+ }
+
+ free(blocks);
+ return EDLIB_STATUS_OK;
+}
+
+
+/**
+ * Takes char query and char target, recognizes alphabet and transforms them into unsigned char sequences
+ * where elements in sequences are not any more letters of alphabet, but their index in alphabet.
+ * Most of internal edlib functions expect such transformed sequences.
+ * This function will allocate queryTransformed and targetTransformed, so make sure to free them when done.
+ * Example:
+ * Original sequences: "ACT" and "CGT".
+ * Alphabet would be recognized as "ACTG". Alphabet length = 4.
+ * Transformed sequences: [0, 1, 2] and [1, 3, 2].
+ * @param [in] queryOriginal
+ * @param [in] queryLength
+ * @param [in] targetOriginal
+ * @param [in] targetLength
+ * @param [out] queryTransformed It will contain values in range [0, alphabet length - 1].
+ * @param [out] targetTransformed It will contain values in range [0, alphabet length - 1].
+ * @return Alphabet as a string of unique characters, where index of each character is its value in transformed
+ * sequences.
+ */
+static char *transformSequences(const char* const queryOriginal, const int queryLength,
+ const char* const targetOriginal, const int targetLength,
+ unsigned char** const queryTransformed,
+ unsigned char** const targetTransformed,
+ int *alphabet_size) {
+ // Alphabet is constructed from letters that are present in sequences.
+ // Each letter is assigned an ordinal number, starting from 0 up to alphabetLength - 1,
+ // and new query and target are created in which letters are replaced with their ordinal numbers.
+ // This query and target are used in all the calculations later.
+ *queryTransformed = malloc(sizeof(unsigned char) * queryLength);
+ *targetTransformed = malloc(sizeof(unsigned char) * targetLength);
+
+ char *alphabet = malloc(MAX_UCHAR+1), *alphabet_cp = alphabet;
+
+ // Alphabet information, it is constructed on fly while transforming sequences.
+ // letterIdx[c] is index of letter c in alphabet.
+ unsigned char letterIdx[MAX_UCHAR + 1];
+ bool inAlphabet[MAX_UCHAR + 1]; // inAlphabet[c] is true if c is in alphabet
+ for (int i = 0; i < MAX_UCHAR + 1; i++) inAlphabet[i] = false;
+
+ for (int i = 0; i < queryLength; i++) {
+ unsigned char c = queryOriginal[i];
+ if (!inAlphabet[c]) {
+ inAlphabet[c] = true;
+ letterIdx[c] = alphabet_cp - alphabet;
+ *alphabet_cp++ = queryOriginal[i];
+ }
+ (*queryTransformed)[i] = letterIdx[c];
+ }
+ for (int i = 0; i < targetLength; i++) {
+ unsigned char c = targetOriginal[i];
+ if (!inAlphabet[c]) {
+ inAlphabet[c] = true;
+ letterIdx[c] = alphabet_cp - alphabet;
+ *alphabet_cp++ = targetOriginal[i];
+ }
+ (*targetTransformed)[i] = letterIdx[c];
+ }
+
+ *alphabet_size = alphabet_cp - alphabet;
+ return alphabet;
+}
+
+
+EdlibAlignConfig edlibNewAlignConfig(int k, EdlibAlignMode mode, EdlibAlignTask task,
+ const EdlibEqualityPair* additionalEqualities,
+ int additionalEqualitiesLength) {
+ EdlibAlignConfig config;
+ config.k = k;
+ config.mode = mode;
+ config.task = task;
+ config.additionalEqualities = additionalEqualities;
+ config.additionalEqualitiesLength = additionalEqualitiesLength;
+ return config;
+}
+
+EdlibAlignConfig edlibDefaultAlignConfig(void) {
+ return edlibNewAlignConfig(-1, EDLIB_MODE_NW, EDLIB_TASK_DISTANCE, NULL, 0);
+}
+
+void edlibFreeAlignResult(EdlibAlignResult result) {
+ if (result.endLocations) free(result.endLocations);
+ if (result.startLocations) free(result.startLocations);
+ if (result.alignment) free(result.alignment);
+}
--- /dev/null
+#ifndef EDLIB_H
+#define EDLIB_H
+
+/**
+ * @file
+ * @author Martin Sosic
+ * @brief Main header file, containing all public functions and structures.
+ */
+
+// Define EDLIB_API macro to properly export symbols
+#ifdef EDLIB_SHARED
+# ifdef _WIN32
+# ifdef EDLIB_BUILD
+# define EDLIB_API __declspec(dllexport)
+# else
+# define EDLIB_API __declspec(dllimport)
+# endif
+# else
+# define EDLIB_API __attribute__ ((visibility ("default")))
+# endif
+#else
+# define EDLIB_API
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+// Status codes
+#define EDLIB_STATUS_OK 0
+#define EDLIB_STATUS_ERROR 1
+
+ /**
+ * Alignment methods - how should Edlib treat gaps before and after query?
+ */
+ typedef enum {
+ /**
+ * Global method. This is the standard method.
+ * Useful when you want to find out how similar is first sequence to second sequence.
+ */
+ EDLIB_MODE_NW,
+ /**
+ * Prefix method. Similar to global method, but with a small twist - gap at query end is not penalized.
+ * What that means is that deleting elements from the end of second sequence is "free"!
+ * For example, if we had "AACT" and "AACTGGC", edit distance would be 0, because removing "GGC" from the end
+ * of second sequence is "free" and does not count into total edit distance. This method is appropriate
+ * when you want to find out how well first sequence fits at the beginning of second sequence.
+ */
+ EDLIB_MODE_SHW,
+ /**
+ * Infix method. Similar as prefix method, but with one more twist - gaps at query end and start are
+ * not penalized. What that means is that deleting elements from the start and end of second sequence is "free"!
+ * For example, if we had ACT and CGACTGAC, edit distance would be 0, because removing CG from the start
+ * and GAC from the end of second sequence is "free" and does not count into total edit distance.
+ * This method is appropriate when you want to find out how well first sequence fits at any part of
+ * second sequence.
+ * For example, if your second sequence was a long text and your first sequence was a sentence from that text,
+ * but slightly scrambled, you could use this method to discover how scrambled it is and where it fits in
+ * that text. In bioinformatics, this method is appropriate for aligning read to a sequence.
+ */
+ EDLIB_MODE_HW
+ } EdlibAlignMode;
+
+ /**
+ * Alignment tasks - what do you want Edlib to do?
+ */
+ typedef enum {
+ EDLIB_TASK_DISTANCE, //!< Find edit distance and end locations.
+ EDLIB_TASK_LOC, //!< Find edit distance, end locations and start locations.
+ EDLIB_TASK_PATH //!< Find edit distance, end locations and start locations and alignment path.
+ } EdlibAlignTask;
+
+ /**
+ * Describes cigar format.
+ * @see http://samtools.github.io/hts-specs/SAMv1.pdf
+ * @see http://drive5.com/usearch/manual/cigar.html
+ */
+ typedef enum {
+ EDLIB_CIGAR_STANDARD, //!< Match: 'M', Insertion: 'I', Deletion: 'D', Mismatch: 'M'.
+ EDLIB_CIGAR_EXTENDED //!< Match: '=', Insertion: 'I', Deletion: 'D', Mismatch: 'X'.
+ } EdlibCigarFormat;
+
+// Edit operations.
+#define EDLIB_EDOP_MATCH 0 //!< Match.
+#define EDLIB_EDOP_INSERT 1 //!< Insertion to target = deletion from query.
+#define EDLIB_EDOP_DELETE 2 //!< Deletion from target = insertion to query.
+#define EDLIB_EDOP_MISMATCH 3 //!< Mismatch.
+
+ /**
+ * @brief Defines two given characters as equal.
+ */
+ typedef struct {
+ char first;
+ char second;
+ } EdlibEqualityPair;
+
+ /**
+ * @brief Configuration object for edlibAlign() function.
+ */
+ typedef struct {
+ /**
+ * Set k to non-negative value to tell edlib that edit distance is not larger than k.
+ * Smaller k can significantly improve speed of computation.
+ * If edit distance is larger than k, edlib will set edit distance to -1.
+ * Set k to negative value and edlib will internally auto-adjust k until score is found.
+ */
+ int k;
+
+ /**
+ * Alignment method.
+ * EDLIB_MODE_NW: global (Needleman-Wunsch)
+ * EDLIB_MODE_SHW: prefix. Gap after query is not penalized.
+ * EDLIB_MODE_HW: infix. Gaps before and after query are not penalized.
+ */
+ EdlibAlignMode mode;
+
+ /**
+ * Alignment task - tells Edlib what to calculate. Less to calculate, faster it is.
+ * EDLIB_TASK_DISTANCE - find edit distance and end locations of optimal alignment paths in target.
+ * EDLIB_TASK_LOC - find edit distance and start and end locations of optimal alignment paths in target.
+ * EDLIB_TASK_PATH - find edit distance, alignment path (and start and end locations of it in target).
+ */
+ EdlibAlignTask task;
+
+ /**
+ * List of pairs of characters, where each pair defines two characters as equal.
+ * This way you can extend edlib's definition of equality (which is that each character is equal only
+ * to itself).
+ * This can be useful if you have some wildcard characters that should match multiple other characters,
+ * or e.g. if you want edlib to be case insensitive.
+ * Can be set to NULL if there are none.
+ */
+ const EdlibEqualityPair* additionalEqualities;
+
+ /**
+ * Number of additional equalities, which is non-negative number.
+ * 0 if there are none.
+ */
+ int additionalEqualitiesLength;
+ } EdlibAlignConfig;
+
+ /**
+ * Helper method for easy construction of configuration object.
+ * @return Configuration object filled with given parameters.
+ */
+ EDLIB_API EdlibAlignConfig edlibNewAlignConfig(
+ int k, EdlibAlignMode mode, EdlibAlignTask task,
+ const EdlibEqualityPair* additionalEqualities,
+ int additionalEqualitiesLength
+ );
+
+ /**
+ * @return Default configuration object, with following defaults:
+ * k = -1, mode = EDLIB_MODE_NW, task = EDLIB_TASK_DISTANCE, no additional equalities.
+ */
+ EDLIB_API EdlibAlignConfig edlibDefaultAlignConfig(void);
+
+
+ /**
+ * Container for results of alignment done by edlibAlign() function.
+ */
+ typedef struct {
+ /**
+ * EDLIB_STATUS_OK or EDLIB_STATUS_ERROR. If error, all other fields will have undefined values.
+ */
+ int status;
+
+ /**
+ * -1 if k is non-negative and edit distance is larger than k.
+ */
+ int editDistance;
+
+ /**
+ * Array of zero-based positions in target where optimal alignment paths end.
+ * If gap after query is penalized, gap counts as part of query (NW), otherwise not.
+ * Set to NULL if edit distance is larger than k.
+ * If you do not free whole result object using edlibFreeAlignResult(), do not forget to use free().
+ */
+ int* endLocations;
+
+ /**
+ * Array of zero-based positions in target where optimal alignment paths start,
+ * they correspond to endLocations.
+ * If gap before query is penalized, gap counts as part of query (NW), otherwise not.
+ * Set to NULL if not calculated or if edit distance is larger than k.
+ * If you do not free whole result object using edlibFreeAlignResult(), do not forget to use free().
+ */
+ int* startLocations;
+
+ /**
+ * Number of end (and start) locations.
+ */
+ int numLocations;
+
+ /**
+ * Alignment is found for first pair of start and end locations.
+ * Set to NULL if not calculated.
+ * Alignment is sequence of numbers: 0, 1, 2, 3.
+ * 0 stands for match.
+ * 1 stands for insertion to target.
+ * 2 stands for insertion to query.
+ * 3 stands for mismatch.
+ * Alignment aligns query to target from beginning of query till end of query.
+ * If gaps are not penalized, they are not in alignment.
+ * If you do not free whole result object using edlibFreeAlignResult(), do not forget to use free().
+ */
+ unsigned char* alignment;
+
+ /**
+ * Length of alignment.
+ */
+ int alignmentLength;
+
+ /**
+ * Number of different characters in query and target together.
+ */
+ int alphabetLength;
+ } EdlibAlignResult;
+
+ /**
+ * Frees memory in EdlibAlignResult that was allocated by edlib.
+ * If you do not use it, make sure to free needed members manually using free().
+ */
+ EDLIB_API void edlibFreeAlignResult(EdlibAlignResult result);
+
+
+ /**
+ * Aligns two sequences (query and target) using edit distance (levenshtein distance).
+ * Through config parameter, this function supports different alignment methods (global, prefix, infix),
+ * as well as different modes of search (tasks).
+ * It always returns edit distance and end locations of optimal alignment in target.
+ * It optionally returns start locations of optimal alignment in target and alignment path,
+ * if you choose appropriate tasks.
+ * @param [in] query First sequence.
+ * @param [in] queryLength Number of characters in first sequence.
+ * @param [in] target Second sequence.
+ * @param [in] targetLength Number of characters in second sequence.
+ * @param [in] config Additional alignment parameters, like alignment method and wanted results.
+ * @return Result of alignment, which can contain edit distance, start and end locations and alignment path.
+ * Make sure to clean up the object using edlibFreeAlignResult() or by manually freeing needed members.
+ */
+ EDLIB_API EdlibAlignResult edlibAlign(
+ const char* query, int queryLength,
+ const char* target, int targetLength,
+ const EdlibAlignConfig config
+ );
+
+
+ /**
+ * Builds cigar string from given alignment sequence.
+ * @param [in] alignment Alignment sequence.
+ * 0 stands for match.
+ * 1 stands for insertion to target.
+ * 2 stands for insertion to query.
+ * 3 stands for mismatch.
+ * @param [in] alignmentLength
+ * @param [in] cigarFormat Cigar will be returned in specified format.
+ * @return Cigar string.
+ * I stands for insertion.
+ * D stands for deletion.
+ * X stands for mismatch. (used only in extended format)
+ * = stands for match. (used only in extended format)
+ * M stands for (mis)match. (used only in standard format)
+ * String is null terminated.
+ * Needed memory is allocated and given pointer is set to it.
+ * Do not forget to free it later using free()!
+ */
+ EDLIB_API char* edlibAlignmentToCigar(
+ const unsigned char* alignment, int alignmentLength,
+ EdlibCigarFormat cigarFormat
+ );
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif // EDLIB_H
/* filter.c -- filter expressions.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <errno.h>
#include <math.h>
#include <sys/types.h>
+#include <stdint.h>
#include <inttypes.h>
#ifndef _WIN32
#include <pwd.h>
#include <htslib/hts_defs.h>
#include <htslib/vcfutils.h>
#include <htslib/kfunc.h>
+#include <htslib/hts_endian.h>
#include "config.h"
#include "filter.h"
#include "bcftools.h"
#if ENABLE_PERL_FILTERS
+// Work around clang warning problems
+# if defined(__clang__)
+# define PERL_GCC_BRACE_GROUPS_FORBIDDEN
+# endif
+
# define filter_t perl_filter_t
# include <EXTERN.h>
# include <perl.h>
char *tag; // for debugging and printout only, VCF tag name
double threshold; // filtering threshold
int is_constant; // the threshold is set
- int hdr_id, tag_type; // BCF header lookup ID and one of BCF_HL_* types
+ int hdr_id, hl_type, ht_type; // BCF header lookup ID and one of BCF_HL_* types and BCF_HT_* types
int idx; // 0-based index to VCF vectors,
// -2: list (e.g. [0,1,2] or [1..3] or [1..] or any field[*], which is equivalent to [0..])
// -3: select indices on the fly based on values in GT
void (*comparator)(struct _token_t *, struct _token_t *, struct _token_t *rtok, bcf1_t *);
void *hash; // test presence of str value in the hash via comparator
regex_t *regex; // precompiled regex for string comparison
+ int iext; // for the use with filter_test_ext(), 1-based index to external values, 0=don't use
// modified on filter evaluation at each VCF line
double *values;
kstring_t str_value;
- int is_str, is_missing; // is_missing is set only for constants, variables are controled via nvalues
+ int is_str, is_missing; // is_missing is set only for constants, variables are controlled via nvalues
int pass_site; // -1 not applicable, 0 fails, >0 pass
uint8_t *pass_samples; // status of individual samples
int nvalues, mvalues; // number of used values: n=0 for missing values, n=1 for scalars, for strings n=str_value.l
char **undef_tag, **used_tag;
int nundef_tag, nused_tag;
int status, exit_on_error;
+ int n_ext; // number of external values to fill via filter_test_ext()
+ int *ext; // types of external values to fill via filter_test_ext()
};
#define TOK_IN 38 // contains, e.g. FILTER~"A"
#define TOK_NOT_IN 39 // does not contain, e.g. FILTER!~"A"
#define TOK_MODULO 40 // %
+#define TOK_EXT 41 // external values set before each filter_test_ext() call, can be one of {},{str},{int},{float}
-// 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
+// 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
// ( ) [ < = > ] ! | & + - * / M m a A O ~ ^ S . l f c p b P i s %
-static int op_prec[] = {0,1,1,5,5,5,5,5,5,2,3, 6, 6, 7, 7, 8, 8, 8, 3, 2, 5, 5, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 7 };
+static int op_prec[] = {0,1,1,5,5,5,5,5,5,2,3, 6, 6, 7, 7, 8, 8, 8, 3, 2, 5, 5, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 7, 0};
#define TOKEN_STRING "x()[<=>]!|&+-*/MmaAO~^S.lfcpis" // this is only for debugging, not maintained diligently
static void cmp_vector_strings(token_t *atok, token_t *btok, token_t *rtok);
+inline static void tok_init_samples(token_t *atok, token_t *btok, token_t *rtok);
+
// Return negative values if it is a function with variable number of arguments
static int filters_next_token(char **str, int *len)
if ( !strncasecmp(tmp,"N_PASS(",7) ) { *len = 6; (*str) += 6; return -TOK_FUNC; }
if ( !strncasecmp(tmp,"F_PASS(",7) ) { *len = 6; (*str) += 6; return -TOK_FUNC; }
if ( !strncasecmp(tmp,"%ILEN",5) ) { *len = 5; return TOK_VAL; } // to be able to distinguish between INFO/ILEN and on-the-fly ILEN
+ if ( !strncasecmp(tmp,"{}",2) ) { *len = 2; return TOK_EXT; }
+ if ( !strncasecmp(tmp,"{STR}",5) ) { *len = 5; return TOK_EXT; }
+ if ( !strncasecmp(tmp,"{INT}",5) ) { *len = 5; return TOK_EXT; }
+ if ( !strncasecmp(tmp,"{FLOAT}",7) ) { *len = 7; return TOK_EXT; }
if ( tmp[0]=='@' ) // file name
{
error("Only ==, !=, ~, and !~ operators are supported for FILTER\n");
return;
}
-static void filters_cmp_id(token_t *atok, token_t *btok, token_t *rtok, bcf1_t *line)
+static void filters_cmp_string_hash(token_t *atok, token_t *btok, token_t *rtok, bcf1_t *line)
{
if ( btok->hash )
{
token_t *tmp = atok; atok = btok; btok = tmp;
}
- if ( atok->hash )
+ if ( rtok->tok_type!=TOK_EQ && rtok->tok_type!=TOK_NE )
+ error("Only == and != operators are supported for strings read from a file\n");
+
+ // INFO
+ if ( !btok->nsamples )
{
- if ( rtok->tok_type!=TOK_EQ && rtok->tok_type!=TOK_NE )
- error("Only == and != operators are supported for strings read from a file\n");
+ // there is only one string value, e.g. STR[1]=@list.txt
+ if ( btok->idx >= 0 )
+ {
+ int ret = btok->str_value.s ? khash_str2int_has_key(atok->hash, btok->str_value.s) : 0;
+ if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
+ rtok->pass_site = ret;
+ return;
+ }
- int ret = khash_str2int_has_key(atok->hash, line->d.id);
+ // there can be multiple comma-separated string values, e.g. STR=@list.txt or STR[*]=@list.txt
+ int ret = 0;
+ char *ptr = btok->str_value.s;
+ while ( *ptr )
+ {
+ char *eptr = ptr + 1;
+ while ( *eptr && *eptr!=',' ) eptr++;
+ char keep = *eptr;
+ *eptr = 0;
+ ret |= khash_str2int_has_key(atok->hash, ptr);
+ *eptr = keep;
+ if ( !keep ) break;
+ ptr = eptr + 1;
+ }
if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
rtok->pass_site = ret;
return;
}
- if ( !btok->str_value.l ) error("Error occurred while evaluating the expression\n");
- if ( rtok->tok_type==TOK_EQ )
- rtok->pass_site = strcmp(btok->str_value.s,line->d.id) ? 0 : 1;
- else if ( rtok->tok_type==TOK_NE )
- rtok->pass_site = strcmp(btok->str_value.s,line->d.id) ? 1 : 0;
- else
+ // FORMAT
+ tok_init_samples(atok, btok, rtok);
+ rtok->pass_site = 0;
+ int i;
+
+ // there is only one string value, e.g. FMT/STR[*:1]=@list.txt
+ if ( btok->idx >= 0 )
+ {
+ for (i=0; i<btok->nsamples; i++)
+ {
+ if ( !rtok->usmpl[i] ) continue;
+ char *str = btok->str_value.s + i*btok->nval1;
+ char keep = str[btok->nval1];
+ str[btok->nval1] = 0;
+ int ret = khash_str2int_has_key(atok->hash, str);
+ str[btok->nval1] = keep;
+ if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
+ rtok->pass_samples[i] = ret;
+ rtok->pass_site |= ret;
+ }
+ return;
+ }
+
+ // there can be multiple comma-separated string values, e.g. FMT/STR=@list.txt
+ for (i=0; i<btok->nsamples; i++)
+ {
+ if ( !rtok->usmpl[i] ) continue;
+ char *str = btok->str_value.s + i*btok->nval1;
+ char keep = str[btok->nval1];
+ str[btok->nval1] = 0;
+
+ // now str contains the block of per-sample comma-separated strings to loop over
+ int ret = 0;
+ char *ptr = str;
+ while ( *ptr )
+ {
+ char *eptr = ptr + 1;
+ while ( *eptr && *eptr!=',' ) eptr++;
+ char keep0 = *eptr;
+ *eptr = 0;
+ ret |= khash_str2int_has_key(atok->hash, ptr);
+ *eptr = keep0;
+ if ( !keep0 ) break;
+ ptr = eptr + 1;
+ }
+ str[btok->nval1] = keep;
+ if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
+ rtok->pass_samples[i] = ret;
+ rtok->pass_site |= ret;
+ }
+}
+static void filters_cmp_id(token_t *atok, token_t *btok, token_t *rtok, bcf1_t *line)
+{
+ if ( btok->hash )
{
- if ( rtok->tok_type!=TOK_LIKE && rtok->tok_type!=TOK_NLIKE )
- error("Only the following operators are supported for querying ID: ==, !=, ~, !~; the operator type %d is not supported (%p %p)\n",
- rtok->tok_type,atok->regex,btok->regex);
+ token_t *tmp = atok; atok = btok; btok = tmp;
+ }
+
+ char *id = line->d.id;
+ int pass = 0;
+
+ while ( id )
+ {
+ char *ep = strchr(id,';');
+ if ( ep ) *ep = 0;
+
+ if ( atok->hash )
+ {
+ if ( rtok->tok_type!=TOK_EQ && rtok->tok_type!=TOK_NE )
+ error("Only == and != operators are supported for strings read from a file\n");
- regex_t *regex = atok->regex ? atok->regex : (btok->regex ? btok->regex : NULL);
- if ( !regex ) error("fixme: regex initialization failed\n");
- rtok->pass_site = regexec(regex,line->d.id, 0,NULL,0) ? 0 : 1;
- if ( rtok->tok_type==TOK_NLIKE ) rtok->pass_site = rtok->pass_site ? 0 : 1;
+ pass = khash_str2int_has_key(atok->hash, id);
+ }
+ else
+ {
+ if ( !btok->str_value.l ) error("Error occurred while evaluating the expression\n");
+
+ if ( rtok->tok_type==TOK_EQ || rtok->tok_type==TOK_NE )
+ pass = strcmp(btok->str_value.s,id) ? 0 : 1;
+ else
+ {
+ if ( rtok->tok_type!=TOK_LIKE && rtok->tok_type!=TOK_NLIKE )
+ error("Only the following operators are supported for querying ID: ==, !=, ~, !~; the operator type %d is not supported (%p %p)\n",
+ rtok->tok_type,atok->regex,btok->regex);
+
+ regex_t *regex = atok->regex ? atok->regex : (btok->regex ? btok->regex : NULL);
+ if ( !regex ) error("fixme: regex initialization failed\n");
+ pass = regexec(regex,id, 0,NULL,0) ? 0 : 1;
+ }
+ }
+ if ( ep )
+ {
+ *ep = ';';
+ id = ep + 1;
+ }
+ if ( pass || !ep ) break;
}
+ if ( rtok->tok_type==TOK_NE || rtok->tok_type==TOK_NE) pass = pass ? 0 : 1;
+ rtok->pass_site = pass;
}
/**
* @line: BCF line
* @info_id: tag ID, as returned by bcf_hdr_id2int
* @ivec: 0-based index to retrieve, -1 when single value is expected
- * @vptr: pointer to memory location of sufficient size to accomodate
+ * @vptr: pointer to memory location of sufficient size to accommodate
* info_id's type
*
* The returned value is -1 if tag is not present, 0 if present but
}
if ( ivec<0 ) ivec = 0;
+ if ( ivec>=info->len) return 0;
- #define BRANCH(type_t, is_missing, is_vector_end, out_type_t) { \
- type_t *p = (type_t *) info->vptr; \
- for (j=0; j<ivec && j<info->len; j++) \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, out_type_t) { \
+ uint8_t *p = info->vptr; \
+ for (j=0; j<ivec; j++) \
{ \
+ type_t val = convert(&p[j * sizeof(type_t)]); \
if ( is_vector_end ) return 0; \
} \
+ type_t val = convert(&p[ivec * sizeof(type_t)]); \
if ( is_missing ) return 0; \
- *((out_type_t*)value) = p[j]; \
+ *((out_type_t*)value) = val; \
return 1; \
}
switch (info->type) {
- case BCF_BT_INT8: BRANCH(int8_t, p[j]==bcf_int8_missing, p[j]==bcf_int8_vector_end, int64_t); break;
- case BCF_BT_INT16: BRANCH(int16_t, p[j]==bcf_int16_missing, p[j]==bcf_int16_vector_end, int64_t); break;
- case BCF_BT_INT32: BRANCH(int32_t, p[j]==bcf_int32_missing, p[j]==bcf_int32_vector_end, int64_t); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(p[j]), bcf_float_is_vector_end(p[j]), double); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, int64_t); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, int64_t); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, int64_t); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), double); break;
default: fprintf(stderr,"todo: type %d\n", info->type); exit(1); break;
}
#undef BRANCH
if ( !(flt->cached_GT.mask[i] & (1<<k)) ) continue;
dst[j++] = src[k];
}
+ if ( !j ) { bcf_double_set_missing(dst[j]); j++; }
for (; j<tok->nval1; j++) bcf_double_set_vector_end(dst[j]);
}
}
dst[j] = src[k];
j++;
}
+ if ( !j ) { bcf_double_set_missing(dst[j]); j++; }
for (; j<tok->nval1; j++) bcf_double_set_vector_end(dst[j]);
}
}
int i, ndim = tok->str_value.m;
int nstr = bcf_get_format_char(flt->hdr, line, tok->tag, &tok->str_value.s, &ndim);
- tok->str_value.m = ndim;
+ tok->str_value.m = tok->str_value.l = ndim;
+ kputc(0,&tok->str_value); // append the nul byte
tok->str_value.l = tok->nvalues = 0;
if ( nstr<0 ) return;
+ if ( tok->idx==-3 && filters_cache_genotypes(flt,line)!=0 ) return;
+
tok->nvalues = tok->str_value.l = nstr;
tok->nval1 = nstr / tok->nsamples;
for (i=0; i<tok->nsamples; i++)
{
if ( !tok->usmpl[i] ) continue;
char *src = tok->str_value.s + i*tok->nval1, *dst = src;
- int ibeg = 0, idx = 0;
+ int ibeg = 0;
+ int idx = 0; // idx-th field
while ( ibeg < tok->nval1 )
{
int iend = ibeg;
while ( iend < tok->nval1 && src[iend] && src[iend]!=',' ) iend++;
int keep = 0;
- if ( tok->idx >= 0 )
+ if ( tok->idx >= 0 ) // single index, given explicitly, e.g. AD[:1]
{
if ( tok->idx==idx ) keep = 1;
}
- else if ( idx < tok->nidxs )
+ else if ( tok->idx == -3 ) // given by GT index, e.g. AD[:GT]
{
- if ( tok->idxs[idx] != 0 ) keep = 1;
+ if ( flt->cached_GT.mask[i] & (1<<idx) ) keep = 1;
+ }
+ else // given as a list, e.g. AD[:0,3]
+ {
+ if ( idx < tok->nidxs )
+ {
+ if ( tok->idxs[idx] != 0 ) keep = 1;
+ }
+ else if ( tok->idxs[tok->nidxs-1] < 0 )
+ keep = 1;
}
- else if ( tok->idxs[tok->nidxs-1] < 0 )
- keep = 1;
-
if ( keep )
{
if ( ibeg!=0 ) memmove(dst, src+ibeg, iend-ibeg+1);
tok->str_value.s = (char*)realloc(tok->str_value.s, tok->str_value.m);
}
-#define BRANCH_INT(type_t,vector_end) \
+#define BRANCH_INT(type_t, convert, vector_end) \
{ \
for (i=0; i<line->n_sample; i++) \
{ \
- type_t *ptr = (type_t*) (fmt->p + i*fmt->size); \
- int is_het = 0, has_ref = 0, missing = 0; \
+ uint8_t *ptr = fmt->p + i*fmt->size; \
+ int is_het = 0, has_ref = 0, missing = 0, jal = 0; \
for (j=0; j<fmt->n; j++) \
{ \
- if ( ptr[j]==vector_end ) break; /* smaller ploidy */ \
- if ( bcf_gt_is_missing(ptr[j]) ) { missing=1; break; } /* missing allele */ \
- int ial = ptr[j]; \
+ type_t ial = convert(&ptr[j * sizeof(type_t)]); \
+ if ( ial==vector_end ) break; /* smaller ploidy */ \
+ if ( bcf_gt_is_missing(ial) ) { missing=1; break; } /* missing allele */ \
if ( bcf_gt_allele(ial)==0 ) has_ref = 1; \
if ( j>0 ) \
{ \
- int jal = ptr[j-1]; \
if ( bcf_gt_allele(ial)!=bcf_gt_allele(jal) ) is_het = 1; \
} \
+ jal = ial; \
} \
char *dst = &tok->str_value.s[nvals1*i]; \
if ( type==4 ) \
} \
}
switch (fmt->type) {
- case BCF_BT_INT8: BRANCH_INT(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH_INT(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH_INT(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH_INT(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH_INT(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH_INT(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: error("The GT type is not lineognised: %d at %s:%"PRId64"\n",fmt->type, bcf_seqname(flt->hdr,line),(int64_t) line->pos+1); break;
}
#undef BRANCH_INT
int i, rlen = strlen(line->d.allele[0]);
for (i=1; i<line->n_allele; i++)
{
+ if ( line->d.allele[i][0]=='<' )
+ {
+ bcf_double_set_missing(tok->values[i-1]);
+ continue;
+ }
int alen = strlen(line->d.allele[i]);
- if ( rlen==alen ) bcf_double_set_missing(tok->values[i-1]);
- else tok->values[i-1] = alen - rlen;
+ tok->values[i-1] = alen - rlen;
}
}
static void filters_set_ref_string(filter_t *flt, bcf1_t *line, token_t *tok)
}
int j,nmissing = 0;
- #define BRANCH(type_t, is_vector_end) { \
+ #define BRANCH(type_t, convert, is_vector_end) { \
for (i=0; i<line->n_sample; i++) \
{ \
- type_t *ptr = (type_t *) (fmt->p + i*fmt->size); \
+ uint8_t *ptr = fmt->p + i*fmt->size; \
for (j=0; j<fmt->n; j++) \
{ \
- if ( ptr[j]==is_vector_end ) break; \
- if ( ptr[j]==bcf_gt_missing ) { nmissing++; break; } \
+ type_t val = convert(&ptr[j * sizeof(type_t)]); \
+ if ( val==is_vector_end ) break; \
+ if ( val==bcf_gt_missing ) { nmissing++; break; } \
} \
} \
}
switch (fmt->type) {
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: fprintf(stderr,"todo: type %d\n", fmt->type); exit(1); break;
}
#undef BRANCH
}
return 1;
}
-static inline double calc_binom(int na, int nb)
-{
- if ( na==0 && nb==0 ) return -1;
- if ( na==nb ) return 1;
-
- // kfunc.h implements kf_betai, which is the regularized beta function P(X<=k/N;p) = I_{1-p}(N-k,k+1)
-
- double pval = na < nb ? kf_betai(nb, na + 1, 0.5) : kf_betai(na, nb + 1, 0.5);
- pval *= 2;
- if ( pval>1 ) pval = 1; // this can happen, machine precision error, eg. kf_betai(1,0,0.5)
-
- return pval;
-}
static int func_binom(filter_t *flt, bcf1_t *line, token_t *rtok, token_t **stack, int nstack)
{
int i, istack = nstack - rtok->nargs;
bcf_double_set_missing(rtok->values[i]);
continue;
}
- rtok->values[i] = calc_binom(vals[idx1],vals[idx2]);
+ rtok->values[i] = calc_binom_two_sided(vals[idx1],vals[idx2],0.5);
if ( rtok->values[i] < 0 )
{
bcf_double_set_missing(rtok->values[i]);
bcf_double_set_missing(rtok->values[i]);
continue;
}
- rtok->values[i] = calc_binom(ptr1[0],ptr2[0]);
+ rtok->values[i] = calc_binom_two_sided(ptr1[0],ptr2[0],0.5);
if ( rtok->values[i] < 0 )
{
bcf_double_set_missing(rtok->values[i]);
bcf_double_set_missing(rtok->values[0]);
else
{
- rtok->values[0] = calc_binom(ptr1[0],ptr2[0]);
+ rtok->values[0] = calc_binom_two_sided(ptr1[0],ptr2[0],0.5);
if ( rtok->values[0] < 0 )
bcf_double_set_missing(rtok->values[0]);
}
return 2;
}
+// A note about comparisons:
+// When setting value by determining index from the genotype, we face the problem
+// of how to interpret truncating arrays. Say we have TAG defined as Number=. and
+// GT:TAG 1/1:0,1,2 0/0:0
+// Then when querying we expect the following expression to evaluate for the second
+// sample as
+// -i 'TAG[1:1]="."' .. true
+// -i 'TAG[1:GT]="."' .. false
+// The problem is that the implementation truncates the number of fields, filling
+// usually fewer than the original number of per-sample values. This is fixed by
+// adding an exception that makes the code aware of this: the GT indexing can be
+// recognised by having tok->idx==-3
#define CMP_VECTORS(atok,btok,_rtok,CMP_OP,missing_logic) \
{ \
token_t *rtok = _rtok; \
double *bptr = btok->values + i*btok->nval1; \
for (j=0; j<atok->nval1; j++) \
{ \
+ if ( atok->idx==-3 && bcf_double_is_vector_end(aptr[j]) ) break; /* explained above */ \
+ if ( btok->idx==-3 && bcf_double_is_vector_end(bptr[j]) ) break; /* explained above */ \
int nmiss = bcf_double_is_missing_or_vector_end(aptr[j]) ? 1 : 0; \
if ( nmiss && !missing_logic[0] ) continue; /* any is missing => result is false */ \
nmiss += (bcf_double_is_missing_or_vector_end(bptr[j]) ? 1 : 0); \
{ \
if ( !rtok->usmpl[i] ) continue; \
double *aptr = atok->values + i*atok->nval1; \
- double *bptr = btok->values + i*btok->nval1; \
+ double *bptr = btok->values; \
for (j=0; j<atok->nval1; j++) \
{ \
+ if ( atok->idx==-3 && bcf_double_is_vector_end(aptr[j]) ) break; /* explained above */ \
int miss = bcf_double_is_missing_or_vector_end(aptr[j]) ? 1 : 0; \
if ( miss && !missing_logic[0] ) continue; /* any is missing => result is false */ \
for (k=0; k<btok->nvalues; k++) \
for (i=0; i<btok->nsamples; i++) \
{ \
if ( !rtok->usmpl[i] ) continue; \
- double *aptr = atok->values + i*atok->nval1; \
+ double *aptr = atok->values; \
double *bptr = btok->values + i*btok->nval1; \
for (j=0; j<btok->nval1; j++) \
{ \
+ if ( atok->idx==-3 && bcf_double_is_vector_end(bptr[j]) ) break; /* explained above */ \
int miss = bcf_double_is_missing_or_vector_end(bptr[j]) ? 1 : 0; \
if ( miss && !missing_logic[0] ) continue; /* any is missing => result is false */ \
for (k=0; k<atok->nvalues; k++) \
return BCF_UN_INFO;
}
+static int filters_init1_ext(filter_t *filter, char *str, int len, token_t *tok)
+{
+ tok->hl_type = -1;
+ tok->ht_type = -1;
+ tok->tok_type = TOK_VAL;
+ tok->hdr_id = -1;
+ tok->pass_site = -1;
+ tok->idx = 0;
+ tok->iext = ++filter->n_ext;
+ filter->ext = realloc(filter->ext,sizeof(*filter->ext)*filter->n_ext);
+ if ( !strncasecmp(str,"{str}",len) ) { tok->ht_type = BCF_HT_STR; tok->is_str = 1; }
+ else if ( !strncasecmp(str,"{int}",len) ) tok->ht_type = BCF_HT_INT;
+ else if ( !strncasecmp(str,"{float}",len) ) tok->ht_type = BCF_HT_REAL;
+ filter->ext[filter->n_ext-1] = tok->ht_type;
+ return 0;
+}
static int filters_init1(filter_t *filter, char *str, int len, token_t *tok)
{
- tok->tag_type = -1;
+ tok->ht_type = -1;
+ tok->hl_type = -1;
tok->tok_type = TOK_VAL;
tok->hdr_id = -1;
tok->pass_site = -1;
memcpy(tok->key,str+1,len-2);
tok->key[len-2] = 0;
tok->is_str = 1;
+ tok->ht_type = BCF_HT_STR;
tok->nvalues = len-2;
if ( !strcmp(".",tok->key) ) tok->is_missing = 1;
return 0;
{
tok->setter = filters_set_qual;
tok->tag = strdup("QUAL");
+ tok->ht_type = BCF_HT_REAL;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->setter = filters_set_type;
tok->tag = strdup("TYPE");
+ tok->ht_type = BCF_HT_STR;
return 0;
}
else if ( !strncasecmp(str,"FILTER",len) || !strncmp(str,"%FILTER",len) /* for backward compatibility */ )
tok->comparator = filters_cmp_filter;
tok->tag = strdup("FILTER");
filter->max_unpack |= BCF_UN_FLT;
- tok->tag_type = BCF_HL_FLT;
+ tok->hl_type = BCF_HL_FLT;
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->comparator = filters_cmp_id;
tok->tag = strdup("ID");
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->setter = &filters_set_chrom;
tok->tag = strdup("CHROM");
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->setter = &filters_set_pos;
tok->tag = strdup("POS");
+ tok->ht_type = BCF_HT_INT;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
tok->setter = &filters_set_ref_string;
tok->is_str = 1;
tok->tag = strdup("REF");
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
tok->setter = &filters_set_alt_string;
tok->is_str = 1;
tok->tag = strdup("ALT");
+ tok->ht_type = BCF_HT_STR;
tok->idxs = (int*) malloc(sizeof(int));
tok->idxs[0] = -1;
tok->nidxs = 1;
{
tok->setter = &filters_set_nalt;
tok->tag = strdup("N_ALT");
+ tok->ht_type = BCF_HT_INT;
return 0;
}
else if ( !strncasecmp(str,"N_SAMPLES",len) )
tok->tok_type = TOK_VAL;
tok->threshold = bcf_hdr_nsamples(filter->hdr);
tok->is_constant = 1;
+ tok->ht_type = BCF_HT_INT;
return 0;
}
else if ( !strncasecmp(str,"N_MISSING",len) )
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_nmissing;
tok->tag = strdup("N_MISSING");
+ tok->ht_type = BCF_HT_INT;
return 0;
}
else if ( !strncasecmp(str,"F_MISSING",len) )
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_nmissing;
tok->tag = strdup("F_MISSING");
+ tok->ht_type = BCF_HT_REAL;
return 0;
}
}
for (i=0; i<tok->nsamples; i++) tok->usmpl[i] = 1;
}
- tok->tag_type = is_fmt ? BCF_HL_FMT : BCF_HL_INFO;
+ tok->hl_type = is_fmt ? BCF_HL_FMT : BCF_HL_INFO;
if ( is_fmt ) filter->max_unpack |= BCF_UN_FMT;
if ( tok->hdr_id>=0 )
{
if ( is_fmt && !strcmp("GT",tmp.s) )
{
tok->setter = &filters_set_genotype_string; tok->is_str = 1;
+ tok->ht_type = BCF_HT_STR;
}
else if ( is_fmt )
{
}
switch ( bcf_hdr_id2type(filter->hdr,BCF_HL_FMT,tok->hdr_id) )
{
- case BCF_HT_INT: tok->setter = &filters_set_format_int; break;
- case BCF_HT_REAL: tok->setter = &filters_set_format_float; break;
- case BCF_HT_STR: tok->setter = &filters_set_format_string; tok->is_str = 1; break;
+ case BCF_HT_INT: tok->setter = &filters_set_format_int; tok->ht_type = BCF_HT_INT; break;
+ case BCF_HT_REAL: tok->setter = &filters_set_format_float; tok->ht_type = BCF_HT_REAL; break;
+ case BCF_HT_STR: tok->setter = &filters_set_format_string; tok->ht_type = BCF_HT_STR; tok->is_str = 1; break;
default: error("[%s:%d %s] FIXME\n", __FILE__,__LINE__,__FUNCTION__);
}
}
else
{
if ( bcf_hdr_id2type(filter->hdr,BCF_HL_INFO,tok->hdr_id) == BCF_HT_FLAG )
+ {
tok->setter = filters_set_info_flag;
+ tok->ht_type = BCF_HT_INT;
+ }
else
{
- if ( bcf_hdr_id2type(filter->hdr,BCF_HL_INFO,tok->hdr_id) == BCF_HT_STR ) tok->is_str = 1;
+ tok->ht_type = bcf_hdr_id2type(filter->hdr,BCF_HL_INFO,tok->hdr_id);
+ if ( tok->ht_type == BCF_HT_STR ) tok->is_str = 1;
if ( bcf_hdr_id2number(filter->hdr,BCF_HL_INFO,tok->hdr_id)==1 )
tok->setter = filters_set_info;
else
{
tok->setter = &filters_set_alt_string;
tok->is_str = 1;
+ tok->ht_type = BCF_HT_STR;
tok->tag = strdup(tmp.s);
free(tmp.s);
filter_add_used_tag(filter,NULL,tok->tag);
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_an;
tok->tag = strdup("AN");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_ac;
tok->tag = strdup("AC");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->max_unpack |= max_ac_an_unpack(filter->hdr);
tok->setter = &filters_set_mac;
tok->tag = strdup("MAC");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->max_unpack |= max_ac_an_unpack(filter->hdr);
tok->setter = &filters_set_af;
tok->tag = strdup("AF");
+ tok->ht_type = BCF_HT_REAL;
free(tmp.s);
return 0;
}
filter->max_unpack |= max_ac_an_unpack(filter->hdr);
tok->setter = &filters_set_maf;
tok->tag = strdup("MAF");
+ tok->ht_type = BCF_HT_REAL;
free(tmp.s);
return 0;
}
filter->max_unpack |= BCF_UN_STR;
tok->setter = &filters_set_ilen;
tok->tag = strdup("ILEN");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->status |= FILTER_ERR_UNKN_TAGS;
filter_add_undef_tag(filter,tmp.s);
}
+ tok->ht_type = BCF_HT_REAL;
}
+ else
+ tok->ht_type = BCF_HT_INT;
tok->is_constant = 1;
if ( tmp.s ) free(tmp.s);
#endif
}
+// A very rudimentary heuristics to determine type, e.g. STR_TAG={} implies {str}.
+// Throws an error on anything more complex and asks for an explicit type.
+static void determine_ext_types(filter_t *filter, int ntok, token_t *tok)
+{
+ int i;
+ for (i=0; i<ntok; i++)
+ {
+ if ( !tok[i].iext || tok[i].ht_type!=-1 ) continue;
+ if ( !i || i+1==ntok ) break; // first or last in the RPN
+ if ( tok[i-1].ht_type==-1 ) break; // previous type not set
+ // todo: check if the next is an operator
+ // todo: case when the order is reversed, {}=TAG
+ tok[i].ht_type = tok[i-1].ht_type;
+ }
+ if ( i!=ntok )
+ error("[%s:%d %s] Error: unable to determine the type, use explicit notation: %s\n",__FILE__,__LINE__,__FUNCTION__,filter->str);
+ for (i=0; i<ntok; i++)
+ {
+ int j = tok[i].iext - 1;
+ if ( j<0 ) continue;
+ if ( filter->ext[j]!=-1 && filter->ext[j]!=tok[i].ht_type )
+ error("[%s:%d %s] FIXME: this should not happen %d vs %d, iext=%d\n",__FILE__,__LINE__,__FUNCTION__,filter->ext[j],tok[i].ht_type,j);
+ filter->ext[j] = tok[i].ht_type;
+ if ( tok[i].ht_type==BCF_HT_STR ) tok[i].is_str = 1;
+ }
+}
+
// Parse filter expression and convert to reverse polish notation. Dijkstra's shunting-yard algorithm
static filter_t *filter_init_(bcf_hdr_t *hdr, const char *str, int exit_on_error)
memset(&ops[nops-1],0,sizeof(token_t));
nops--;
}
+ else if ( ret==TOK_EXT ) // external value
+ {
+ nout++;
+ hts_expand0(token_t, nout, mout, out);
+ filters_init1_ext(filter, tmp, len, &out[nout-1]);
+ tmp += len;
+ }
else if ( ret!=TOK_VAL ) // one of the operators
{
// detect unary minus: replace -value with -1*(value)
hts_expand0(token_t, nops, mops, ops);
ops[nops-1].tok_type = ret;
}
- else if ( !len )
+ else if ( !len ) // all tokes read or an error
{
if ( *tmp && !isspace(*tmp) ) error("Could not parse the expression: [%s]\n", str);
break; // all tokens read
}
- else // annotation name or filtering value
+ else // TOK_VAL: annotation name or value
{
nout++;
hts_expand0(token_t, nout, mout, out);
nops--;
}
+ if ( filter->status != FILTER_OK )
+ {
+ if ( mops ) free(ops);
+ filter->filters = out;
+ filter->nfilters = nout;
+ return filter;
+ }
+
+ // Determine types of external variables from the context
+ determine_ext_types(filter,nout,out);
+
// In the special cases of TYPE and FILTER the BCF header IDs are yet unknown. Walk through the
// list of operators and convert the strings (e.g. "PASS") to BCF ids. The string value token must be
// just before or after the FILTER token and they must be followed with a comparison operator.
- // At this point we also initialize regex expressions which, in RPN, must preceed the LIKE/NLIKE operator.
+ // At this point we also initialize regex expressions which, in RPN, must precede the LIKE/NLIKE operator.
// Additionally, treat "." as missing value rather than a string in numeric equalities; that
// @file is only used with ID; etc.
// This code is fragile: improve me.
{
int j = out[i+1].tok_type==TOK_VAL ? i+1 : i-1;
if ( out[j].comparator!=filters_cmp_id )
- error("Error: could not parse the expression. Note that the \"@file_name\" syntax can be currently used with ID column only.\n");
+ {
+ if ( out[j].comparator ) error("Error: could not parse the expression with \"@file_name\" syntax (possible todo)\n");
+ out[j].comparator = filters_cmp_string_hash;
+ }
}
if ( out[i].tok_type==TOK_OR || out[i].tok_type==TOK_OR_VEC )
out[i].func = vector_logic_or;
int set_missing = 0;
if ( out[k].hdr_id>0 )
{
- int type = bcf_hdr_id2type(filter->hdr,out[k].tag_type,out[k].hdr_id);
+ int type = bcf_hdr_id2type(filter->hdr,out[k].hl_type,out[k].hdr_id);
if ( type==BCF_HT_INT ) set_missing = 1;
else if ( type==BCF_HT_REAL ) set_missing = 1;
}
if ( !out[i].tag ) continue;
if ( out[i].setter==filters_set_type )
{
- if ( i+1==nout ) error("Could not parse the expression: %s\n", filter->str);
+ if ( i+1==nout || !out[i+1].key )
+ error("Could not parse the expression: %s\n", filter->str);
int itok, ival;
if ( out[i+1].tok_type==TOK_EQ || out[i+1].tok_type==TOK_NE ) ival = i - 1, itok = i + 1;
else if ( out[i+1].tok_type==TOK_LIKE || out[i+1].tok_type==TOK_NLIKE ) ival = i - 1, itok = i + 1;
else if ( !strcasecmp(out[ival].key,"r") ) { out[i].setter = filters_set_genotype2; out[ival].key[0]='r'; out[ival].key[1]=0; } // r
continue;
}
- if ( out[i].tag_type==BCF_HL_FLT )
+ if ( out[i].hl_type==BCF_HL_FLT )
{
if ( i+1==nout ) error("Could not parse the expression: %s\n", filter->str);
int itok = i, ival;
}
for (i=0; i<filter->nundef_tag; i++) free(filter->undef_tag[i]);
for (i=0; i<filter->nused_tag; i++) free(filter->used_tag[i]);
+ free(filter->ext);
free(filter->undef_tag);
free(filter->used_tag);
free(filter->cached_GT.buf);
free(filter);
}
+int filter_test_ext(filter_t *filter, bcf1_t *rec, const uint8_t **samples, const void **ext)
+{
+ if ( !filter->n_ext )
+ return filter_test(filter,rec,samples);
+
+ int i;
+ for (i=0; i<filter->nfilters; i++)
+ {
+ token_t *tok = &filter->filters[i];
+ if ( !tok->iext ) continue;
+ if ( !ext[tok->iext-1] )
+ {
+ tok->is_missing = 1;
+ tok->nvalues = 0;
+ if ( filter->ext[tok->iext-1]==BCF_HT_STR ) tok->str_value.l = 0;
+ continue;
+ }
+ tok->is_missing = 0;
+ tok->nvalues = 1;
+ if ( filter->ext[tok->iext-1]==BCF_HT_STR )
+ {
+ tok->str_value.l = 0;
+ kputs((const char*)ext[tok->iext-1],&tok->str_value);
+ tok->nvalues = tok->str_value.l;
+ }
+ else if ( filter->ext[tok->iext-1]==BCF_HT_INT ) tok->values[0] = *((const int*)ext[tok->iext-1]);
+ else if ( filter->ext[tok->iext-1]==BCF_HT_REAL ) tok->values[0] = *((const float*)ext[tok->iext-1]);
+ }
+ return filter_test(filter,rec,samples);
+}
+
int filter_test(filter_t *filter, bcf1_t *line, const uint8_t **samples)
{
if ( filter->status != FILTER_OK ) error("Error: the caller did not check the filter status\n");
{
return flt->max_unpack;
}
-
+const int *filter_ext_types(filter_t *filter, int *n_ext)
+{
+ *n_ext = filter->n_ext;
+ return filter->ext;
+}
const double *filter_get_doubles(filter_t *filter, int *nval, int *nval1)
{
token_t *tok = filter->flt_stack[0];
/* filter.c -- filter expressions.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <errno.h>
#include <math.h>
#include <sys/types.h>
+#include <stdint.h>
#include <inttypes.h>
#ifndef _WIN32
#include <pwd.h>
#include <htslib/hts_defs.h>
#include <htslib/vcfutils.h>
#include <htslib/kfunc.h>
+#include <htslib/hts_endian.h>
#include "config.h"
#include "filter.h"
#include "bcftools.h"
#if ENABLE_PERL_FILTERS
+// Work around clang warning problems
+# if defined(__clang__)
+# define PERL_GCC_BRACE_GROUPS_FORBIDDEN
+# endif
+
# define filter_t perl_filter_t
# include <EXTERN.h>
# include <perl.h>
char *tag; // for debugging and printout only, VCF tag name
double threshold; // filtering threshold
int is_constant; // the threshold is set
- int hdr_id, tag_type; // BCF header lookup ID and one of BCF_HL_* types
+ int hdr_id, hl_type, ht_type; // BCF header lookup ID and one of BCF_HL_* types and BCF_HT_* types
int idx; // 0-based index to VCF vectors,
// -2: list (e.g. [0,1,2] or [1..3] or [1..] or any field[*], which is equivalent to [0..])
// -3: select indices on the fly based on values in GT
void (*comparator)(struct _token_t *, struct _token_t *, struct _token_t *rtok, bcf1_t *);
void *hash; // test presence of str value in the hash via comparator
regex_t *regex; // precompiled regex for string comparison
+ int iext; // for the use with filter_test_ext(), 1-based index to external values, 0=don't use
// modified on filter evaluation at each VCF line
double *values;
kstring_t str_value;
- int is_str, is_missing; // is_missing is set only for constants, variables are controled via nvalues
+ int is_str, is_missing; // is_missing is set only for constants, variables are controlled via nvalues
int pass_site; // -1 not applicable, 0 fails, >0 pass
uint8_t *pass_samples; // status of individual samples
int nvalues, mvalues; // number of used values: n=0 for missing values, n=1 for scalars, for strings n=str_value.l
char **undef_tag, **used_tag;
int nundef_tag, nused_tag;
int status, exit_on_error;
+ int n_ext; // number of external values to fill via filter_test_ext()
+ int *ext; // types of external values to fill via filter_test_ext()
};
#define TOK_IN 38 // contains, e.g. FILTER~"A"
#define TOK_NOT_IN 39 // does not contain, e.g. FILTER!~"A"
#define TOK_MODULO 40 // %
+#define TOK_EXT 41 // external values set before each filter_test_ext() call, can be one of {},{str},{int},{float}
-// 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
+// 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
// ( ) [ < = > ] ! | & + - * / M m a A O ~ ^ S . l f c p b P i s %
-static int op_prec[] = {0,1,1,5,5,5,5,5,5,2,3, 6, 6, 7, 7, 8, 8, 8, 3, 2, 5, 5, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 7 };
+static int op_prec[] = {0,1,1,5,5,5,5,5,5,2,3, 6, 6, 7, 7, 8, 8, 8, 3, 2, 5, 5, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 7, 0};
#define TOKEN_STRING "x()[<=>]!|&+-*/MmaAO~^S.lfcpis" // this is only for debugging, not maintained diligently
static void cmp_vector_strings(token_t *atok, token_t *btok, token_t *rtok);
+inline static void tok_init_samples(token_t *atok, token_t *btok, token_t *rtok);
+
// Return negative values if it is a function with variable number of arguments
static int filters_next_token(char **str, int *len)
if ( !strncasecmp(tmp,"N_PASS(",7) ) { *len = 6; (*str) += 6; return -TOK_FUNC; }
if ( !strncasecmp(tmp,"F_PASS(",7) ) { *len = 6; (*str) += 6; return -TOK_FUNC; }
if ( !strncasecmp(tmp,"%ILEN",5) ) { *len = 5; return TOK_VAL; } // to be able to distinguish between INFO/ILEN and on-the-fly ILEN
+ if ( !strncasecmp(tmp,"{}",2) ) { *len = 2; return TOK_EXT; }
+ if ( !strncasecmp(tmp,"{STR}",5) ) { *len = 5; return TOK_EXT; }
+ if ( !strncasecmp(tmp,"{INT}",5) ) { *len = 5; return TOK_EXT; }
+ if ( !strncasecmp(tmp,"{FLOAT}",7) ) { *len = 7; return TOK_EXT; }
if ( tmp[0]=='@' ) // file name
{
error("Only ==, !=, ~, and !~ operators are supported for FILTER\n");
return;
}
-static void filters_cmp_id(token_t *atok, token_t *btok, token_t *rtok, bcf1_t *line)
+static void filters_cmp_string_hash(token_t *atok, token_t *btok, token_t *rtok, bcf1_t *line)
{
if ( btok->hash )
{
token_t *tmp = atok; atok = btok; btok = tmp;
}
- if ( atok->hash )
+ if ( rtok->tok_type!=TOK_EQ && rtok->tok_type!=TOK_NE )
+ error("Only == and != operators are supported for strings read from a file\n");
+
+ // INFO
+ if ( !btok->nsamples )
{
- if ( rtok->tok_type!=TOK_EQ && rtok->tok_type!=TOK_NE )
- error("Only == and != operators are supported for strings read from a file\n");
+ // there is only one string value, e.g. STR[1]=@list.txt
+ if ( btok->idx >= 0 )
+ {
+ int ret = btok->str_value.s ? khash_str2int_has_key(atok->hash, btok->str_value.s) : 0;
+ if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
+ rtok->pass_site = ret;
+ return;
+ }
- int ret = khash_str2int_has_key(atok->hash, line->d.id);
+ // there can be multiple comma-separated string values, e.g. STR=@list.txt or STR[*]=@list.txt
+ int ret = 0;
+ char *ptr = btok->str_value.s;
+ while ( *ptr )
+ {
+ char *eptr = ptr + 1;
+ while ( *eptr && *eptr!=',' ) eptr++;
+ char keep = *eptr;
+ *eptr = 0;
+ ret |= khash_str2int_has_key(atok->hash, ptr);
+ *eptr = keep;
+ if ( !keep ) break;
+ ptr = eptr + 1;
+ }
if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
rtok->pass_site = ret;
return;
}
- if ( !btok->str_value.l ) error("Error occurred while evaluating the expression\n");
- if ( rtok->tok_type==TOK_EQ )
- rtok->pass_site = strcmp(btok->str_value.s,line->d.id) ? 0 : 1;
- else if ( rtok->tok_type==TOK_NE )
- rtok->pass_site = strcmp(btok->str_value.s,line->d.id) ? 1 : 0;
- else
+ // FORMAT
+ tok_init_samples(atok, btok, rtok);
+ rtok->pass_site = 0;
+ int i;
+
+ // there is only one string value, e.g. FMT/STR[*:1]=@list.txt
+ if ( btok->idx >= 0 )
+ {
+ for (i=0; i<btok->nsamples; i++)
+ {
+ if ( !rtok->usmpl[i] ) continue;
+ char *str = btok->str_value.s + i*btok->nval1;
+ char keep = str[btok->nval1];
+ str[btok->nval1] = 0;
+ int ret = khash_str2int_has_key(atok->hash, str);
+ str[btok->nval1] = keep;
+ if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
+ rtok->pass_samples[i] = ret;
+ rtok->pass_site |= ret;
+ }
+ return;
+ }
+
+ // there can be multiple comma-separated string values, e.g. FMT/STR=@list.txt
+ for (i=0; i<btok->nsamples; i++)
+ {
+ if ( !rtok->usmpl[i] ) continue;
+ char *str = btok->str_value.s + i*btok->nval1;
+ char keep = str[btok->nval1];
+ str[btok->nval1] = 0;
+
+ // now str contains the block of per-sample comma-separated strings to loop over
+ int ret = 0;
+ char *ptr = str;
+ while ( *ptr )
+ {
+ char *eptr = ptr + 1;
+ while ( *eptr && *eptr!=',' ) eptr++;
+ char keep0 = *eptr;
+ *eptr = 0;
+ ret |= khash_str2int_has_key(atok->hash, ptr);
+ *eptr = keep0;
+ if ( !keep0 ) break;
+ ptr = eptr + 1;
+ }
+ str[btok->nval1] = keep;
+ if ( rtok->tok_type==TOK_NE ) ret = ret ? 0 : 1;
+ rtok->pass_samples[i] = ret;
+ rtok->pass_site |= ret;
+ }
+}
+static void filters_cmp_id(token_t *atok, token_t *btok, token_t *rtok, bcf1_t *line)
+{
+ if ( btok->hash )
{
- if ( rtok->tok_type!=TOK_LIKE && rtok->tok_type!=TOK_NLIKE )
- error("Only the following operators are supported for querying ID: ==, !=, ~, !~; the operator type %d is not supported (%p %p)\n",
- rtok->tok_type,atok->regex,btok->regex);
+ token_t *tmp = atok; atok = btok; btok = tmp;
+ }
+
+ char *id = line->d.id;
+ int pass = 0;
+
+ while ( id )
+ {
+ char *ep = strchr(id,';');
+ if ( ep ) *ep = 0;
+
+ if ( atok->hash )
+ {
+ if ( rtok->tok_type!=TOK_EQ && rtok->tok_type!=TOK_NE )
+ error("Only == and != operators are supported for strings read from a file\n");
- regex_t *regex = atok->regex ? atok->regex : (btok->regex ? btok->regex : NULL);
- if ( !regex ) error("fixme: regex initialization failed\n");
- rtok->pass_site = regexec(regex,line->d.id, 0,NULL,0) ? 0 : 1;
- if ( rtok->tok_type==TOK_NLIKE ) rtok->pass_site = rtok->pass_site ? 0 : 1;
+ pass = khash_str2int_has_key(atok->hash, id);
+ }
+ else
+ {
+ if ( !btok->str_value.l ) error("Error occurred while evaluating the expression\n");
+
+ if ( rtok->tok_type==TOK_EQ || rtok->tok_type==TOK_NE )
+ pass = strcmp(btok->str_value.s,id) ? 0 : 1;
+ else
+ {
+ if ( rtok->tok_type!=TOK_LIKE && rtok->tok_type!=TOK_NLIKE )
+ error("Only the following operators are supported for querying ID: ==, !=, ~, !~; the operator type %d is not supported (%p %p)\n",
+ rtok->tok_type,atok->regex,btok->regex);
+
+ regex_t *regex = atok->regex ? atok->regex : (btok->regex ? btok->regex : NULL);
+ if ( !regex ) error("fixme: regex initialization failed\n");
+ pass = regexec(regex,id, 0,NULL,0) ? 0 : 1;
+ }
+ }
+ if ( ep )
+ {
+ *ep = ';';
+ id = ep + 1;
+ }
+ if ( pass || !ep ) break;
}
+ if ( rtok->tok_type==TOK_NE || rtok->tok_type==TOK_NE) pass = pass ? 0 : 1;
+ rtok->pass_site = pass;
}
/**
* @line: BCF line
* @info_id: tag ID, as returned by bcf_hdr_id2int
* @ivec: 0-based index to retrieve, -1 when single value is expected
- * @vptr: pointer to memory location of sufficient size to accomodate
+ * @vptr: pointer to memory location of sufficient size to accommodate
* info_id's type
*
* The returned value is -1 if tag is not present, 0 if present but
}
if ( ivec<0 ) ivec = 0;
+ if ( ivec>=info->len) return 0;
- #define BRANCH(type_t, is_missing, is_vector_end, out_type_t) { \
- type_t *p = (type_t *) info->vptr; \
- for (j=0; j<ivec && j<info->len; j++) \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, out_type_t) { \
+ uint8_t *p = info->vptr; \
+ for (j=0; j<ivec; j++) \
{ \
+ type_t val = convert(&p[j * sizeof(type_t)]); \
if ( is_vector_end ) return 0; \
} \
+ type_t val = convert(&p[ivec * sizeof(type_t)]); \
if ( is_missing ) return 0; \
- *((out_type_t*)value) = p[j]; \
+ *((out_type_t*)value) = val; \
return 1; \
}
switch (info->type) {
- case BCF_BT_INT8: BRANCH(int8_t, p[j]==bcf_int8_missing, p[j]==bcf_int8_vector_end, int64_t); break;
- case BCF_BT_INT16: BRANCH(int16_t, p[j]==bcf_int16_missing, p[j]==bcf_int16_vector_end, int64_t); break;
- case BCF_BT_INT32: BRANCH(int32_t, p[j]==bcf_int32_missing, p[j]==bcf_int32_vector_end, int64_t); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(p[j]), bcf_float_is_vector_end(p[j]), double); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, int64_t); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, int64_t); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, int64_t); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), double); break;
default: fprintf(bcftools_stderr,"todo: type %d\n", info->type); bcftools_exit(1); break;
}
#undef BRANCH
if ( !(flt->cached_GT.mask[i] & (1<<k)) ) continue;
dst[j++] = src[k];
}
+ if ( !j ) { bcf_double_set_missing(dst[j]); j++; }
for (; j<tok->nval1; j++) bcf_double_set_vector_end(dst[j]);
}
}
dst[j] = src[k];
j++;
}
+ if ( !j ) { bcf_double_set_missing(dst[j]); j++; }
for (; j<tok->nval1; j++) bcf_double_set_vector_end(dst[j]);
}
}
int i, ndim = tok->str_value.m;
int nstr = bcf_get_format_char(flt->hdr, line, tok->tag, &tok->str_value.s, &ndim);
- tok->str_value.m = ndim;
+ tok->str_value.m = tok->str_value.l = ndim;
+ kputc(0,&tok->str_value); // append the nul byte
tok->str_value.l = tok->nvalues = 0;
if ( nstr<0 ) return;
+ if ( tok->idx==-3 && filters_cache_genotypes(flt,line)!=0 ) return;
+
tok->nvalues = tok->str_value.l = nstr;
tok->nval1 = nstr / tok->nsamples;
for (i=0; i<tok->nsamples; i++)
{
if ( !tok->usmpl[i] ) continue;
char *src = tok->str_value.s + i*tok->nval1, *dst = src;
- int ibeg = 0, idx = 0;
+ int ibeg = 0;
+ int idx = 0; // idx-th field
while ( ibeg < tok->nval1 )
{
int iend = ibeg;
while ( iend < tok->nval1 && src[iend] && src[iend]!=',' ) iend++;
int keep = 0;
- if ( tok->idx >= 0 )
+ if ( tok->idx >= 0 ) // single index, given explicitly, e.g. AD[:1]
{
if ( tok->idx==idx ) keep = 1;
}
- else if ( idx < tok->nidxs )
+ else if ( tok->idx == -3 ) // given by GT index, e.g. AD[:GT]
{
- if ( tok->idxs[idx] != 0 ) keep = 1;
+ if ( flt->cached_GT.mask[i] & (1<<idx) ) keep = 1;
+ }
+ else // given as a list, e.g. AD[:0,3]
+ {
+ if ( idx < tok->nidxs )
+ {
+ if ( tok->idxs[idx] != 0 ) keep = 1;
+ }
+ else if ( tok->idxs[tok->nidxs-1] < 0 )
+ keep = 1;
}
- else if ( tok->idxs[tok->nidxs-1] < 0 )
- keep = 1;
-
if ( keep )
{
if ( ibeg!=0 ) memmove(dst, src+ibeg, iend-ibeg+1);
tok->str_value.s = (char*)realloc(tok->str_value.s, tok->str_value.m);
}
-#define BRANCH_INT(type_t,vector_end) \
+#define BRANCH_INT(type_t, convert, vector_end) \
{ \
for (i=0; i<line->n_sample; i++) \
{ \
- type_t *ptr = (type_t*) (fmt->p + i*fmt->size); \
- int is_het = 0, has_ref = 0, missing = 0; \
+ uint8_t *ptr = fmt->p + i*fmt->size; \
+ int is_het = 0, has_ref = 0, missing = 0, jal = 0; \
for (j=0; j<fmt->n; j++) \
{ \
- if ( ptr[j]==vector_end ) break; /* smaller ploidy */ \
- if ( bcf_gt_is_missing(ptr[j]) ) { missing=1; break; } /* missing allele */ \
- int ial = ptr[j]; \
+ type_t ial = convert(&ptr[j * sizeof(type_t)]); \
+ if ( ial==vector_end ) break; /* smaller ploidy */ \
+ if ( bcf_gt_is_missing(ial) ) { missing=1; break; } /* missing allele */ \
if ( bcf_gt_allele(ial)==0 ) has_ref = 1; \
if ( j>0 ) \
{ \
- int jal = ptr[j-1]; \
if ( bcf_gt_allele(ial)!=bcf_gt_allele(jal) ) is_het = 1; \
} \
+ jal = ial; \
} \
char *dst = &tok->str_value.s[nvals1*i]; \
if ( type==4 ) \
} \
}
switch (fmt->type) {
- case BCF_BT_INT8: BRANCH_INT(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH_INT(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH_INT(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH_INT(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH_INT(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH_INT(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: error("The GT type is not lineognised: %d at %s:%"PRId64"\n",fmt->type, bcf_seqname(flt->hdr,line),(int64_t) line->pos+1); break;
}
#undef BRANCH_INT
int i, rlen = strlen(line->d.allele[0]);
for (i=1; i<line->n_allele; i++)
{
+ if ( line->d.allele[i][0]=='<' )
+ {
+ bcf_double_set_missing(tok->values[i-1]);
+ continue;
+ }
int alen = strlen(line->d.allele[i]);
- if ( rlen==alen ) bcf_double_set_missing(tok->values[i-1]);
- else tok->values[i-1] = alen - rlen;
+ tok->values[i-1] = alen - rlen;
}
}
static void filters_set_ref_string(filter_t *flt, bcf1_t *line, token_t *tok)
}
int j,nmissing = 0;
- #define BRANCH(type_t, is_vector_end) { \
+ #define BRANCH(type_t, convert, is_vector_end) { \
for (i=0; i<line->n_sample; i++) \
{ \
- type_t *ptr = (type_t *) (fmt->p + i*fmt->size); \
+ uint8_t *ptr = fmt->p + i*fmt->size; \
for (j=0; j<fmt->n; j++) \
{ \
- if ( ptr[j]==is_vector_end ) break; \
- if ( ptr[j]==bcf_gt_missing ) { nmissing++; break; } \
+ type_t val = convert(&ptr[j * sizeof(type_t)]); \
+ if ( val==is_vector_end ) break; \
+ if ( val==bcf_gt_missing ) { nmissing++; break; } \
} \
} \
}
switch (fmt->type) {
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: fprintf(bcftools_stderr,"todo: type %d\n", fmt->type); bcftools_exit(1); break;
}
#undef BRANCH
}
return 1;
}
-static inline double calc_binom(int na, int nb)
-{
- if ( na==0 && nb==0 ) return -1;
- if ( na==nb ) return 1;
-
- // kfunc.h implements kf_betai, which is the regularized beta function P(X<=k/N;p) = I_{1-p}(N-k,k+1)
-
- double pval = na < nb ? kf_betai(nb, na + 1, 0.5) : kf_betai(na, nb + 1, 0.5);
- pval *= 2;
- if ( pval>1 ) pval = 1; // this can happen, machine precision error, eg. kf_betai(1,0,0.5)
-
- return pval;
-}
static int func_binom(filter_t *flt, bcf1_t *line, token_t *rtok, token_t **stack, int nstack)
{
int i, istack = nstack - rtok->nargs;
bcf_double_set_missing(rtok->values[i]);
continue;
}
- rtok->values[i] = calc_binom(vals[idx1],vals[idx2]);
+ rtok->values[i] = calc_binom_two_sided(vals[idx1],vals[idx2],0.5);
if ( rtok->values[i] < 0 )
{
bcf_double_set_missing(rtok->values[i]);
bcf_double_set_missing(rtok->values[i]);
continue;
}
- rtok->values[i] = calc_binom(ptr1[0],ptr2[0]);
+ rtok->values[i] = calc_binom_two_sided(ptr1[0],ptr2[0],0.5);
if ( rtok->values[i] < 0 )
{
bcf_double_set_missing(rtok->values[i]);
bcf_double_set_missing(rtok->values[0]);
else
{
- rtok->values[0] = calc_binom(ptr1[0],ptr2[0]);
+ rtok->values[0] = calc_binom_two_sided(ptr1[0],ptr2[0],0.5);
if ( rtok->values[0] < 0 )
bcf_double_set_missing(rtok->values[0]);
}
return 2;
}
+// A note about comparisons:
+// When setting value by determining index from the genotype, we face the problem
+// of how to interpret truncating arrays. Say we have TAG defined as Number=. and
+// GT:TAG 1/1:0,1,2 0/0:0
+// Then when querying we expect the following expression to evaluate for the second
+// sample as
+// -i 'TAG[1:1]="."' .. true
+// -i 'TAG[1:GT]="."' .. false
+// The problem is that the implementation truncates the number of fields, filling
+// usually fewer than the original number of per-sample values. This is fixed by
+// adding an exception that makes the code aware of this: the GT indexing can be
+// recognised by having tok->idx==-3
#define CMP_VECTORS(atok,btok,_rtok,CMP_OP,missing_logic) \
{ \
token_t *rtok = _rtok; \
double *bptr = btok->values + i*btok->nval1; \
for (j=0; j<atok->nval1; j++) \
{ \
+ if ( atok->idx==-3 && bcf_double_is_vector_end(aptr[j]) ) break; /* explained above */ \
+ if ( btok->idx==-3 && bcf_double_is_vector_end(bptr[j]) ) break; /* explained above */ \
int nmiss = bcf_double_is_missing_or_vector_end(aptr[j]) ? 1 : 0; \
if ( nmiss && !missing_logic[0] ) continue; /* any is missing => result is false */ \
nmiss += (bcf_double_is_missing_or_vector_end(bptr[j]) ? 1 : 0); \
{ \
if ( !rtok->usmpl[i] ) continue; \
double *aptr = atok->values + i*atok->nval1; \
- double *bptr = btok->values + i*btok->nval1; \
+ double *bptr = btok->values; \
for (j=0; j<atok->nval1; j++) \
{ \
+ if ( atok->idx==-3 && bcf_double_is_vector_end(aptr[j]) ) break; /* explained above */ \
int miss = bcf_double_is_missing_or_vector_end(aptr[j]) ? 1 : 0; \
if ( miss && !missing_logic[0] ) continue; /* any is missing => result is false */ \
for (k=0; k<btok->nvalues; k++) \
for (i=0; i<btok->nsamples; i++) \
{ \
if ( !rtok->usmpl[i] ) continue; \
- double *aptr = atok->values + i*atok->nval1; \
+ double *aptr = atok->values; \
double *bptr = btok->values + i*btok->nval1; \
for (j=0; j<btok->nval1; j++) \
{ \
+ if ( atok->idx==-3 && bcf_double_is_vector_end(bptr[j]) ) break; /* explained above */ \
int miss = bcf_double_is_missing_or_vector_end(bptr[j]) ? 1 : 0; \
if ( miss && !missing_logic[0] ) continue; /* any is missing => result is false */ \
for (k=0; k<atok->nvalues; k++) \
return BCF_UN_INFO;
}
+static int filters_init1_ext(filter_t *filter, char *str, int len, token_t *tok)
+{
+ tok->hl_type = -1;
+ tok->ht_type = -1;
+ tok->tok_type = TOK_VAL;
+ tok->hdr_id = -1;
+ tok->pass_site = -1;
+ tok->idx = 0;
+ tok->iext = ++filter->n_ext;
+ filter->ext = realloc(filter->ext,sizeof(*filter->ext)*filter->n_ext);
+ if ( !strncasecmp(str,"{str}",len) ) { tok->ht_type = BCF_HT_STR; tok->is_str = 1; }
+ else if ( !strncasecmp(str,"{int}",len) ) tok->ht_type = BCF_HT_INT;
+ else if ( !strncasecmp(str,"{float}",len) ) tok->ht_type = BCF_HT_REAL;
+ filter->ext[filter->n_ext-1] = tok->ht_type;
+ return 0;
+}
static int filters_init1(filter_t *filter, char *str, int len, token_t *tok)
{
- tok->tag_type = -1;
+ tok->ht_type = -1;
+ tok->hl_type = -1;
tok->tok_type = TOK_VAL;
tok->hdr_id = -1;
tok->pass_site = -1;
memcpy(tok->key,str+1,len-2);
tok->key[len-2] = 0;
tok->is_str = 1;
+ tok->ht_type = BCF_HT_STR;
tok->nvalues = len-2;
if ( !strcmp(".",tok->key) ) tok->is_missing = 1;
return 0;
{
tok->setter = filters_set_qual;
tok->tag = strdup("QUAL");
+ tok->ht_type = BCF_HT_REAL;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->setter = filters_set_type;
tok->tag = strdup("TYPE");
+ tok->ht_type = BCF_HT_STR;
return 0;
}
else if ( !strncasecmp(str,"FILTER",len) || !strncmp(str,"%FILTER",len) /* for backward compatibility */ )
tok->comparator = filters_cmp_filter;
tok->tag = strdup("FILTER");
filter->max_unpack |= BCF_UN_FLT;
- tok->tag_type = BCF_HL_FLT;
+ tok->hl_type = BCF_HL_FLT;
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->comparator = filters_cmp_id;
tok->tag = strdup("ID");
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->setter = &filters_set_chrom;
tok->tag = strdup("CHROM");
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
{
tok->setter = &filters_set_pos;
tok->tag = strdup("POS");
+ tok->ht_type = BCF_HT_INT;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
tok->setter = &filters_set_ref_string;
tok->is_str = 1;
tok->tag = strdup("REF");
+ tok->ht_type = BCF_HT_STR;
filter_add_used_tag(filter,NULL,tok->tag);
return 0;
}
tok->setter = &filters_set_alt_string;
tok->is_str = 1;
tok->tag = strdup("ALT");
+ tok->ht_type = BCF_HT_STR;
tok->idxs = (int*) malloc(sizeof(int));
tok->idxs[0] = -1;
tok->nidxs = 1;
{
tok->setter = &filters_set_nalt;
tok->tag = strdup("N_ALT");
+ tok->ht_type = BCF_HT_INT;
return 0;
}
else if ( !strncasecmp(str,"N_SAMPLES",len) )
tok->tok_type = TOK_VAL;
tok->threshold = bcf_hdr_nsamples(filter->hdr);
tok->is_constant = 1;
+ tok->ht_type = BCF_HT_INT;
return 0;
}
else if ( !strncasecmp(str,"N_MISSING",len) )
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_nmissing;
tok->tag = strdup("N_MISSING");
+ tok->ht_type = BCF_HT_INT;
return 0;
}
else if ( !strncasecmp(str,"F_MISSING",len) )
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_nmissing;
tok->tag = strdup("F_MISSING");
+ tok->ht_type = BCF_HT_REAL;
return 0;
}
}
for (i=0; i<tok->nsamples; i++) tok->usmpl[i] = 1;
}
- tok->tag_type = is_fmt ? BCF_HL_FMT : BCF_HL_INFO;
+ tok->hl_type = is_fmt ? BCF_HL_FMT : BCF_HL_INFO;
if ( is_fmt ) filter->max_unpack |= BCF_UN_FMT;
if ( tok->hdr_id>=0 )
{
if ( is_fmt && !strcmp("GT",tmp.s) )
{
tok->setter = &filters_set_genotype_string; tok->is_str = 1;
+ tok->ht_type = BCF_HT_STR;
}
else if ( is_fmt )
{
}
switch ( bcf_hdr_id2type(filter->hdr,BCF_HL_FMT,tok->hdr_id) )
{
- case BCF_HT_INT: tok->setter = &filters_set_format_int; break;
- case BCF_HT_REAL: tok->setter = &filters_set_format_float; break;
- case BCF_HT_STR: tok->setter = &filters_set_format_string; tok->is_str = 1; break;
+ case BCF_HT_INT: tok->setter = &filters_set_format_int; tok->ht_type = BCF_HT_INT; break;
+ case BCF_HT_REAL: tok->setter = &filters_set_format_float; tok->ht_type = BCF_HT_REAL; break;
+ case BCF_HT_STR: tok->setter = &filters_set_format_string; tok->ht_type = BCF_HT_STR; tok->is_str = 1; break;
default: error("[%s:%d %s] FIXME\n", __FILE__,__LINE__,__FUNCTION__);
}
}
else
{
if ( bcf_hdr_id2type(filter->hdr,BCF_HL_INFO,tok->hdr_id) == BCF_HT_FLAG )
+ {
tok->setter = filters_set_info_flag;
+ tok->ht_type = BCF_HT_INT;
+ }
else
{
- if ( bcf_hdr_id2type(filter->hdr,BCF_HL_INFO,tok->hdr_id) == BCF_HT_STR ) tok->is_str = 1;
+ tok->ht_type = bcf_hdr_id2type(filter->hdr,BCF_HL_INFO,tok->hdr_id);
+ if ( tok->ht_type == BCF_HT_STR ) tok->is_str = 1;
if ( bcf_hdr_id2number(filter->hdr,BCF_HL_INFO,tok->hdr_id)==1 )
tok->setter = filters_set_info;
else
{
tok->setter = &filters_set_alt_string;
tok->is_str = 1;
+ tok->ht_type = BCF_HT_STR;
tok->tag = strdup(tmp.s);
free(tmp.s);
filter_add_used_tag(filter,NULL,tok->tag);
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_an;
tok->tag = strdup("AN");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->max_unpack |= BCF_UN_FMT;
tok->setter = &filters_set_ac;
tok->tag = strdup("AC");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->max_unpack |= max_ac_an_unpack(filter->hdr);
tok->setter = &filters_set_mac;
tok->tag = strdup("MAC");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->max_unpack |= max_ac_an_unpack(filter->hdr);
tok->setter = &filters_set_af;
tok->tag = strdup("AF");
+ tok->ht_type = BCF_HT_REAL;
free(tmp.s);
return 0;
}
filter->max_unpack |= max_ac_an_unpack(filter->hdr);
tok->setter = &filters_set_maf;
tok->tag = strdup("MAF");
+ tok->ht_type = BCF_HT_REAL;
free(tmp.s);
return 0;
}
filter->max_unpack |= BCF_UN_STR;
tok->setter = &filters_set_ilen;
tok->tag = strdup("ILEN");
+ tok->ht_type = BCF_HT_INT;
free(tmp.s);
return 0;
}
filter->status |= FILTER_ERR_UNKN_TAGS;
filter_add_undef_tag(filter,tmp.s);
}
+ tok->ht_type = BCF_HT_REAL;
}
+ else
+ tok->ht_type = BCF_HT_INT;
tok->is_constant = 1;
if ( tmp.s ) free(tmp.s);
#endif
}
+// A very rudimentary heuristics to determine type, e.g. STR_TAG={} implies {str}.
+// Throws an error on anything more complex and asks for an explicit type.
+static void determine_ext_types(filter_t *filter, int ntok, token_t *tok)
+{
+ int i;
+ for (i=0; i<ntok; i++)
+ {
+ if ( !tok[i].iext || tok[i].ht_type!=-1 ) continue;
+ if ( !i || i+1==ntok ) break; // first or last in the RPN
+ if ( tok[i-1].ht_type==-1 ) break; // previous type not set
+ // todo: check if the next is an operator
+ // todo: case when the order is reversed, {}=TAG
+ tok[i].ht_type = tok[i-1].ht_type;
+ }
+ if ( i!=ntok )
+ error("[%s:%d %s] Error: unable to determine the type, use explicit notation: %s\n",__FILE__,__LINE__,__FUNCTION__,filter->str);
+ for (i=0; i<ntok; i++)
+ {
+ int j = tok[i].iext - 1;
+ if ( j<0 ) continue;
+ if ( filter->ext[j]!=-1 && filter->ext[j]!=tok[i].ht_type )
+ error("[%s:%d %s] FIXME: this should not happen %d vs %d, iext=%d\n",__FILE__,__LINE__,__FUNCTION__,filter->ext[j],tok[i].ht_type,j);
+ filter->ext[j] = tok[i].ht_type;
+ if ( tok[i].ht_type==BCF_HT_STR ) tok[i].is_str = 1;
+ }
+}
+
// Parse filter expression and convert to reverse polish notation. Dijkstra's shunting-yard algorithm
static filter_t *filter_init_(bcf_hdr_t *hdr, const char *str, int exit_on_error)
memset(&ops[nops-1],0,sizeof(token_t));
nops--;
}
+ else if ( ret==TOK_EXT ) // external value
+ {
+ nout++;
+ hts_expand0(token_t, nout, mout, out);
+ filters_init1_ext(filter, tmp, len, &out[nout-1]);
+ tmp += len;
+ }
else if ( ret!=TOK_VAL ) // one of the operators
{
// detect unary minus: replace -value with -1*(value)
hts_expand0(token_t, nops, mops, ops);
ops[nops-1].tok_type = ret;
}
- else if ( !len )
+ else if ( !len ) // all tokes read or an error
{
if ( *tmp && !isspace(*tmp) ) error("Could not parse the expression: [%s]\n", str);
break; // all tokens read
}
- else // annotation name or filtering value
+ else // TOK_VAL: annotation name or value
{
nout++;
hts_expand0(token_t, nout, mout, out);
nops--;
}
+ if ( filter->status != FILTER_OK )
+ {
+ if ( mops ) free(ops);
+ filter->filters = out;
+ filter->nfilters = nout;
+ return filter;
+ }
+
+ // Determine types of external variables from the context
+ determine_ext_types(filter,nout,out);
+
// In the special cases of TYPE and FILTER the BCF header IDs are yet unknown. Walk through the
// list of operators and convert the strings (e.g. "PASS") to BCF ids. The string value token must be
// just before or after the FILTER token and they must be followed with a comparison operator.
- // At this point we also initialize regex expressions which, in RPN, must preceed the LIKE/NLIKE operator.
+ // At this point we also initialize regex expressions which, in RPN, must precede the LIKE/NLIKE operator.
// Additionally, treat "." as missing value rather than a string in numeric equalities; that
// @file is only used with ID; etc.
// This code is fragile: improve me.
{
int j = out[i+1].tok_type==TOK_VAL ? i+1 : i-1;
if ( out[j].comparator!=filters_cmp_id )
- error("Error: could not parse the expression. Note that the \"@file_name\" syntax can be currently used with ID column only.\n");
+ {
+ if ( out[j].comparator ) error("Error: could not parse the expression with \"@file_name\" syntax (possible todo)\n");
+ out[j].comparator = filters_cmp_string_hash;
+ }
}
if ( out[i].tok_type==TOK_OR || out[i].tok_type==TOK_OR_VEC )
out[i].func = vector_logic_or;
int set_missing = 0;
if ( out[k].hdr_id>0 )
{
- int type = bcf_hdr_id2type(filter->hdr,out[k].tag_type,out[k].hdr_id);
+ int type = bcf_hdr_id2type(filter->hdr,out[k].hl_type,out[k].hdr_id);
if ( type==BCF_HT_INT ) set_missing = 1;
else if ( type==BCF_HT_REAL ) set_missing = 1;
}
if ( !out[i].tag ) continue;
if ( out[i].setter==filters_set_type )
{
- if ( i+1==nout ) error("Could not parse the expression: %s\n", filter->str);
+ if ( i+1==nout || !out[i+1].key )
+ error("Could not parse the expression: %s\n", filter->str);
int itok, ival;
if ( out[i+1].tok_type==TOK_EQ || out[i+1].tok_type==TOK_NE ) ival = i - 1, itok = i + 1;
else if ( out[i+1].tok_type==TOK_LIKE || out[i+1].tok_type==TOK_NLIKE ) ival = i - 1, itok = i + 1;
else if ( !strcasecmp(out[ival].key,"r") ) { out[i].setter = filters_set_genotype2; out[ival].key[0]='r'; out[ival].key[1]=0; } // r
continue;
}
- if ( out[i].tag_type==BCF_HL_FLT )
+ if ( out[i].hl_type==BCF_HL_FLT )
{
if ( i+1==nout ) error("Could not parse the expression: %s\n", filter->str);
int itok = i, ival;
}
for (i=0; i<filter->nundef_tag; i++) free(filter->undef_tag[i]);
for (i=0; i<filter->nused_tag; i++) free(filter->used_tag[i]);
+ free(filter->ext);
free(filter->undef_tag);
free(filter->used_tag);
free(filter->cached_GT.buf);
free(filter);
}
+int filter_test_ext(filter_t *filter, bcf1_t *rec, const uint8_t **samples, const void **ext)
+{
+ if ( !filter->n_ext )
+ return filter_test(filter,rec,samples);
+
+ int i;
+ for (i=0; i<filter->nfilters; i++)
+ {
+ token_t *tok = &filter->filters[i];
+ if ( !tok->iext ) continue;
+ if ( !ext[tok->iext-1] )
+ {
+ tok->is_missing = 1;
+ tok->nvalues = 0;
+ if ( filter->ext[tok->iext-1]==BCF_HT_STR ) tok->str_value.l = 0;
+ continue;
+ }
+ tok->is_missing = 0;
+ tok->nvalues = 1;
+ if ( filter->ext[tok->iext-1]==BCF_HT_STR )
+ {
+ tok->str_value.l = 0;
+ kputs((const char*)ext[tok->iext-1],&tok->str_value);
+ tok->nvalues = tok->str_value.l;
+ }
+ else if ( filter->ext[tok->iext-1]==BCF_HT_INT ) tok->values[0] = *((const int*)ext[tok->iext-1]);
+ else if ( filter->ext[tok->iext-1]==BCF_HT_REAL ) tok->values[0] = *((const float*)ext[tok->iext-1]);
+ }
+ return filter_test(filter,rec,samples);
+}
+
int filter_test(filter_t *filter, bcf1_t *line, const uint8_t **samples)
{
if ( filter->status != FILTER_OK ) error("Error: the caller did not check the filter status\n");
{
return flt->max_unpack;
}
-
+const int *filter_ext_types(filter_t *filter, int *n_ext)
+{
+ *n_ext = filter->n_ext;
+ return filter->ext;
+}
const double *filter_get_doubles(filter_t *filter, int *nval, int *nval1)
{
token_t *tok = filter->flt_stack[0];
/* filter.h -- filter expressions.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
/**
* @hdr: BCF header file
- * @str: see the bcftools filter command help for description
- *
+ * @str: see the bcftools filter command help for description.
+ * See also the extended usage described in filter_test_ext(),
+ * intended for programmatic access
* Same as filter_parse() but exits on errors
*/
filter_t *filter_init(bcf_hdr_t *hdr, const char *str);
*/
int filter_test(filter_t *filter, bcf1_t *rec, const uint8_t **samples);
+/**
+ * filter_test_ext() - same as filter_test(), but sets some of the terms
+ * on the fly. An expression initialized with, say,
+ * "STR_TAG={} | INT_TAG={} | FLT_TAG={}" takes three
+ * additional pointer arguments which are expected to point to memory
+ * area occupied by the appropriate type, see also filter_ext_types().
+ * The type determination is not fool-proof, in such case the type can
+ * be given explicitly as eg "TAG={str}".
+ * @ext: array of size 'n_ext' occupied with pointers to the data types
+ * inferred from the expression given at the time of initialization.
+ * The pointers set to NULL will be treated as if missing value "."
+ * was given.
+ * @n_ext: the size of 'ext' array
+ */
+int filter_test_ext(filter_t *filter, bcf1_t *rec, const uint8_t **samples, const void **ext);
+
/**
* filter_set_samples() - restrict filtering expression to samples.
* Call after filter_init().
*/
const double *filter_get_doubles(filter_t *filter, int *nval, int *nval1);
-void filter_expression_info(FILE *fp);
int filter_max_unpack(filter_t *filter);
+/**
+ * filter_ext_types() - returns the number and BCF_HT_* types of external values
+ * found in the filtering expression
+ */
+const int *filter_ext_types(filter_t *filter, int *n_ext);
+
/**
* Same as filter_init() but may not exit on some type of errors. The caller
* must check if the returned value is not NULL and if the consequent call
THE SOFTWARE.
*/
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <inttypes.h>
+#include <string.h>
+#include <strings.h>
+#include <htslib/hts.h>
+#include <htslib/khash.h>
+#include <htslib/khash_str2int.h>
+#include <htslib/kseq.h>
+#include <htslib/bgzf.h>
+#include <errno.h>
+#include "bcftools.h"
+#include "regidx.h"
#include "gff.h"
/*
uint32_t beg;
uint32_t end;
uint32_t trid;
- uint32_t strand:1; // STRAND_REV,STRAND_FWD
+ uint32_t strand:2; // STRAND_{REV,FWD,UNK}
uint32_t phase:2; // 0, 1, 2, or 3 for unknown
- uint32_t iseq:29;
+ uint32_t iseq:28;
}
ftr_t;
// associate with transcript id
gff_id_register(&gff->tscript_ids, aux->parent, aux->parent_end, &ftr->trid);
- if ( ftr->strand==-1 && gff->verbosity > 0 )
+ if ( ftr->strand==STRAND_UNK && gff->verbosity > 0 )
{
if ( !gff->warned.unknown_strand || gff->verbosity > 1 )
fprintf(stderr,"Warning: Ignoring GFF feature with unknown strand .. %s\n",line);
gff->warned.unknown_strand++;
}
- if ( ftr->phase==-1 && gff->verbosity > 0 )
+ if ( ftr->phase==CDS_PHASE_UNKN && gff->verbosity > 0 )
{
if ( !gff->warned.unknown_phase|| gff->verbosity > 1 )
fprintf(stderr,"Warning: Ignoring GFF feature with unknown phase .. %s\n",line);
gene->name = strdup(aux->gene_ids.str[gene_id]); // Name=<GeneName> field is not present, use the gene ID instead
}
-// Returns 0 for exons,CDS,UTRs to indiciate these need to be pruned later and regidx built on them,
-// or -1 to indiciate the structure needs not be saved (either because of an error or because saved
+// Returns 0 for exons,CDS,UTRs to indicate these need to be pruned later and regidx built on them,
+// or -1 to indicate the structure needs not be saved (either because of an error or because saved
// as transcript or gene.)
static int gff_parse_line(gff_t *gff, char *line, ftr_t *ftr)
{
ftr->strand = -1;
if ( *ss == '+' ) ftr->strand = STRAND_FWD;
else if ( *ss == '-' ) ftr->strand = STRAND_REV;
+ else ftr->strand = STRAND_UNK;
ss += 2;
// 8th column: phase (codon offset)
- ftr->phase = -1;
+ ftr->phase = CDS_PHASE_UNKN;
if ( *ss == '0' ) ftr->phase = 0;
else if ( *ss == '1' ) ftr->phase = 1;
else if ( *ss == '2' ) ftr->phase = 2;
if ( phase!=len%3 )
{
if ( !gff->force )
- error("Error: GFF3 assumption failed for transcript %s, CDS=%d: phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
+ error("Error: GFF3 assumption failed for transcript %s, CDS=%"PRIu32": phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
if ( gff->verbosity > 0 )
{
if ( !gff->warned.wrong_phase || gff->verbosity > 1 )
- fprintf(stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%d: phase!=len%%3 (phase=%d, len=%d)\n",
+ fprintf(stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%"PRIu32": phase!=len%%3 (phase=%d, len=%d)\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
gff->warned.wrong_phase++;
}
}
if ( !tscript_ok ) continue; // skip this transcript
}
- else
+ else if ( tr->strand==STRAND_REV )
{
if ( tr->cds[tr->ncds-1]->phase != CDS_PHASE_UNKN )
{
if ( phase!=len%3 )
{
if ( !gff->force )
- error("Error: GFF3 assumption failed for transcript %s, CDS=%d: phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
+ error("Error: GFF3 assumption failed for transcript %s, CDS=%"PRIu32": phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
if ( gff->verbosity > 0 )
{
if ( !gff->warned.wrong_phase || gff->verbosity > 1 )
- fprintf(stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%d: phase!=len%%3 (phase=%d, len=%d)\n",
+ fprintf(stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%"PRIu32": phase!=len%%3 (phase=%d, len=%d)\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
gff->warned.wrong_phase++;
}
}
if ( !tscript_ok ) continue; // skip this transcript
}
+ else
+ continue; // unknown strand
// set len. At the same check that CDS within a transcript do not overlap
len = 0;
i--;
}
}
- else
+ else if ( tr->strand==STRAND_REV )
{
i = 0;
while ( i<tr->ncds && len%3 )
gf_gene_t *gene = (gf_gene_t*) kh_val(gff->init.gid2gene, k);
char *gene_id = gff->init.gene_ids.str[gene->id];
str.l = 0;
- ksprintf(&str,"%s\t.\tgene\t%d\t%d\t.\t%c\t.\tID=%s;Name=%s;used=%d\n",gff->init.seq[gene->iseq],gene->beg+1,gene->end+1,gene->strand==STRAND_FWD?'+':'-',gene_id,gene->name,gene->used);
+ ksprintf(&str,"%s\t.\tgene\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tID=%s;Name=%s;used=%d\n",gff->init.seq[gene->iseq],gene->beg+1,gene->end+1,gene->strand==STRAND_FWD?'+':(gene->strand==STRAND_REV?'-':'.'),gene_id,gene->name,gene->used);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
char *gene_id = gff->init.gene_ids.str[tr->gene->id];
const char *type = tr->type==GF_PROTEIN_CODING ? "mRNA" : gf_type2gff_string(tr->type);
str.l = 0;
- ksprintf(&str,"%s\t.\t%s\t%d\t%d\t.\t%c\t.\tID=%s;Parent=%s;biotype=%s;used=%d\n",itr->seq,type,itr->beg+1,itr->end+1,tr->strand==STRAND_FWD?'+':'-',gff->tscript_ids.str[tr->id],gene_id,gf_type2gff_string(tr->type),tr->used);
+ ksprintf(&str,"%s\t.\t%s\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tID=%s;Parent=%s;biotype=%s;used=%d\n",itr->seq,type,itr->beg+1,itr->end+1,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),gff->tscript_ids.str[tr->id],gene_id,gf_type2gff_string(tr->type),tr->used);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
gf_cds_t *cds = regitr_payload(itr,gf_cds_t*);
gf_tscript_t *tr = cds->tr;
str.l = 0;
- ksprintf(&str,"%s\t.\tCDS\t%d\t%d\t.\t%c\t%c\tParent=%s\n",itr->seq,cds->beg+1,cds->beg+cds->len,tr->strand==STRAND_FWD?'+':'-',cds->phase==3?'.':cds->phase+(int)'0',gff->tscript_ids.str[tr->id]);
+ ksprintf(&str,"%s\t.\tCDS\t%"PRIu32"\t%"PRIu32"\t.\t%c\t%c\tParent=%s\n",itr->seq,cds->beg+1,cds->beg+cds->len,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),cds->phase==3?'.':cds->phase+(int)'0',gff->tscript_ids.str[tr->id]);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
gf_utr_t *utr = regitr_payload(itr,gf_utr_t*);
gf_tscript_t *tr = utr->tr;
str.l = 0;
- ksprintf(&str,"%s\t.\t%s_prime_UTR\t%d\t%d\t.\t%c\t.\tParent=%s\n",itr->seq,utr->which==prime3?"three":"five",utr->beg+1,utr->end+1,tr->strand==STRAND_FWD?'+':'-',gff->tscript_ids.str[tr->id]);
+ ksprintf(&str,"%s\t.\t%s_prime_UTR\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tParent=%s\n",itr->seq,utr->which==prime3?"three":"five",utr->beg+1,utr->end+1,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),gff->tscript_ids.str[tr->id]);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
gf_exon_t *exon = regitr_payload(itr,gf_exon_t*);
gf_tscript_t *tr = exon->tr;
str.l = 0;
- ksprintf(&str,"%s\t.\texon\t%d\t%d\t.\t%c\t.\tParent=%s\n",itr->seq,exon->beg+1,exon->end+1,tr->strand==STRAND_FWD?'+':'-',gff->tscript_ids.str[tr->id]);
+ ksprintf(&str,"%s\t.\texon\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tParent=%s\n",itr->seq,exon->beg+1,exon->end+1,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),gff->tscript_ids.str[tr->id]);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
else if ( ftr->type==GF_UTR5 ) register_utr(gff, ftr);
else if ( ftr->type==GF_UTR3 ) register_utr(gff, ftr);
else
- error("something: %s\t%d\t%d\t%s\t%s\n", aux->seq[ftr->iseq],ftr->beg+1,ftr->end+1,gff->tscript_ids.str[ftr->trid],gf_type2gff_string(ftr->type));
+ error("something: %s\t%"PRIu32"\t%"PRIu32"\t%s\t%s\n", aux->seq[ftr->iseq],ftr->beg+1,ftr->end+1,gff->tscript_ids.str[ftr->trid],gf_type2gff_string(ftr->type));
}
tscript_init_cds(gff);
INC_NWARN(wrong_phase);
INC_NWARN(overlapping_cds);
if ( nwarn > 0 )
- fprintf(stderr,"Warning: %d warnings were supressed, run with `--verbose 2` to see them all\n",nwarn);
+ fprintf(stderr,"Warning: %d warnings were suppressed, increase verbosity to see them all\n",nwarn);
}
if ( gff->dump_fname ) gff_dump(gff, gff->dump_fname);
THE SOFTWARE.
*/
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <inttypes.h>
+#include <string.h>
+#include <strings.h>
+#include <htslib/hts.h>
+#include <htslib/khash.h>
+#include <htslib/khash_str2int.h>
+#include <htslib/kseq.h>
+#include <htslib/bgzf.h>
+#include <errno.h>
+#include "bcftools.h"
+#include "regidx.h"
#include "gff.h"
/*
uint32_t beg;
uint32_t end;
uint32_t trid;
- uint32_t strand:1; // STRAND_REV,STRAND_FWD
+ uint32_t strand:2; // STRAND_{REV,FWD,UNK}
uint32_t phase:2; // 0, 1, 2, or 3 for unknown
- uint32_t iseq:29;
+ uint32_t iseq:28;
}
ftr_t;
// associate with transcript id
gff_id_register(&gff->tscript_ids, aux->parent, aux->parent_end, &ftr->trid);
- if ( ftr->strand==-1 && gff->verbosity > 0 )
+ if ( ftr->strand==STRAND_UNK && gff->verbosity > 0 )
{
if ( !gff->warned.unknown_strand || gff->verbosity > 1 )
fprintf(bcftools_stderr,"Warning: Ignoring GFF feature with unknown strand .. %s\n",line);
gff->warned.unknown_strand++;
}
- if ( ftr->phase==-1 && gff->verbosity > 0 )
+ if ( ftr->phase==CDS_PHASE_UNKN && gff->verbosity > 0 )
{
if ( !gff->warned.unknown_phase|| gff->verbosity > 1 )
fprintf(bcftools_stderr,"Warning: Ignoring GFF feature with unknown phase .. %s\n",line);
gene->name = strdup(aux->gene_ids.str[gene_id]); // Name=<GeneName> field is not present, use the gene ID instead
}
-// Returns 0 for exons,CDS,UTRs to indiciate these need to be pruned later and regidx built on them,
-// or -1 to indiciate the structure needs not be saved (either because of an error or because saved
+// Returns 0 for exons,CDS,UTRs to indicate these need to be pruned later and regidx built on them,
+// or -1 to indicate the structure needs not be saved (either because of an error or because saved
// as transcript or gene.)
static int gff_parse_line(gff_t *gff, char *line, ftr_t *ftr)
{
ftr->strand = -1;
if ( *ss == '+' ) ftr->strand = STRAND_FWD;
else if ( *ss == '-' ) ftr->strand = STRAND_REV;
+ else ftr->strand = STRAND_UNK;
ss += 2;
// 8th column: phase (codon offset)
- ftr->phase = -1;
+ ftr->phase = CDS_PHASE_UNKN;
if ( *ss == '0' ) ftr->phase = 0;
else if ( *ss == '1' ) ftr->phase = 1;
else if ( *ss == '2' ) ftr->phase = 2;
if ( phase!=len%3 )
{
if ( !gff->force )
- error("Error: GFF3 assumption failed for transcript %s, CDS=%d: phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
+ error("Error: GFF3 assumption failed for transcript %s, CDS=%"PRIu32": phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
if ( gff->verbosity > 0 )
{
if ( !gff->warned.wrong_phase || gff->verbosity > 1 )
- fprintf(bcftools_stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%d: phase!=len%%3 (phase=%d, len=%d)\n",
+ fprintf(bcftools_stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%"PRIu32": phase!=len%%3 (phase=%d, len=%d)\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
gff->warned.wrong_phase++;
}
}
if ( !tscript_ok ) continue; // skip this transcript
}
- else
+ else if ( tr->strand==STRAND_REV )
{
if ( tr->cds[tr->ncds-1]->phase != CDS_PHASE_UNKN )
{
if ( phase!=len%3 )
{
if ( !gff->force )
- error("Error: GFF3 assumption failed for transcript %s, CDS=%d: phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
+ error("Error: GFF3 assumption failed for transcript %s, CDS=%"PRIu32": phase!=len%%3 (phase=%d, len=%d). Use the --force option to proceed anyway (at your own risk).\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
if ( gff->verbosity > 0 )
{
if ( !gff->warned.wrong_phase || gff->verbosity > 1 )
- fprintf(bcftools_stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%d: phase!=len%%3 (phase=%d, len=%d)\n",
+ fprintf(bcftools_stderr,"Warning: The GFF has inconsistent phase column in transcript %s, skipping. CDS pos=%"PRIu32": phase!=len%%3 (phase=%d, len=%d)\n",
gff->tscript_ids.str[tr->id],tr->cds[i]->beg+1,phase,len);
gff->warned.wrong_phase++;
}
}
if ( !tscript_ok ) continue; // skip this transcript
}
+ else
+ continue; // unknown strand
// set len. At the same check that CDS within a transcript do not overlap
len = 0;
i--;
}
}
- else
+ else if ( tr->strand==STRAND_REV )
{
i = 0;
while ( i<tr->ncds && len%3 )
gf_gene_t *gene = (gf_gene_t*) kh_val(gff->init.gid2gene, k);
char *gene_id = gff->init.gene_ids.str[gene->id];
str.l = 0;
- ksprintf(&str,"%s\t.\tgene\t%d\t%d\t.\t%c\t.\tID=%s;Name=%s;used=%d\n",gff->init.seq[gene->iseq],gene->beg+1,gene->end+1,gene->strand==STRAND_FWD?'+':'-',gene_id,gene->name,gene->used);
+ ksprintf(&str,"%s\t.\tgene\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tID=%s;Name=%s;used=%d\n",gff->init.seq[gene->iseq],gene->beg+1,gene->end+1,gene->strand==STRAND_FWD?'+':(gene->strand==STRAND_REV?'-':'.'),gene_id,gene->name,gene->used);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
char *gene_id = gff->init.gene_ids.str[tr->gene->id];
const char *type = tr->type==GF_PROTEIN_CODING ? "mRNA" : gf_type2gff_string(tr->type);
str.l = 0;
- ksprintf(&str,"%s\t.\t%s\t%d\t%d\t.\t%c\t.\tID=%s;Parent=%s;biotype=%s;used=%d\n",itr->seq,type,itr->beg+1,itr->end+1,tr->strand==STRAND_FWD?'+':'-',gff->tscript_ids.str[tr->id],gene_id,gf_type2gff_string(tr->type),tr->used);
+ ksprintf(&str,"%s\t.\t%s\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tID=%s;Parent=%s;biotype=%s;used=%d\n",itr->seq,type,itr->beg+1,itr->end+1,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),gff->tscript_ids.str[tr->id],gene_id,gf_type2gff_string(tr->type),tr->used);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
gf_cds_t *cds = regitr_payload(itr,gf_cds_t*);
gf_tscript_t *tr = cds->tr;
str.l = 0;
- ksprintf(&str,"%s\t.\tCDS\t%d\t%d\t.\t%c\t%c\tParent=%s\n",itr->seq,cds->beg+1,cds->beg+cds->len,tr->strand==STRAND_FWD?'+':'-',cds->phase==3?'.':cds->phase+(int)'0',gff->tscript_ids.str[tr->id]);
+ ksprintf(&str,"%s\t.\tCDS\t%"PRIu32"\t%"PRIu32"\t.\t%c\t%c\tParent=%s\n",itr->seq,cds->beg+1,cds->beg+cds->len,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),cds->phase==3?'.':cds->phase+(int)'0',gff->tscript_ids.str[tr->id]);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
gf_utr_t *utr = regitr_payload(itr,gf_utr_t*);
gf_tscript_t *tr = utr->tr;
str.l = 0;
- ksprintf(&str,"%s\t.\t%s_prime_UTR\t%d\t%d\t.\t%c\t.\tParent=%s\n",itr->seq,utr->which==prime3?"three":"five",utr->beg+1,utr->end+1,tr->strand==STRAND_FWD?'+':'-',gff->tscript_ids.str[tr->id]);
+ ksprintf(&str,"%s\t.\t%s_prime_UTR\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tParent=%s\n",itr->seq,utr->which==prime3?"three":"five",utr->beg+1,utr->end+1,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),gff->tscript_ids.str[tr->id]);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
gf_exon_t *exon = regitr_payload(itr,gf_exon_t*);
gf_tscript_t *tr = exon->tr;
str.l = 0;
- ksprintf(&str,"%s\t.\texon\t%d\t%d\t.\t%c\t.\tParent=%s\n",itr->seq,exon->beg+1,exon->end+1,tr->strand==STRAND_FWD?'+':'-',gff->tscript_ids.str[tr->id]);
+ ksprintf(&str,"%s\t.\texon\t%"PRIu32"\t%"PRIu32"\t.\t%c\t.\tParent=%s\n",itr->seq,exon->beg+1,exon->end+1,tr->strand==STRAND_FWD?'+':(tr->strand==STRAND_REV?'-':'.'),gff->tscript_ids.str[tr->id]);
if ( bgzf_write(out, str.s, str.l) != str.l ) error("Error writing %s: %s\n", fname, strerror(errno));
}
regitr_destroy(itr);
else if ( ftr->type==GF_UTR5 ) register_utr(gff, ftr);
else if ( ftr->type==GF_UTR3 ) register_utr(gff, ftr);
else
- error("something: %s\t%d\t%d\t%s\t%s\n", aux->seq[ftr->iseq],ftr->beg+1,ftr->end+1,gff->tscript_ids.str[ftr->trid],gf_type2gff_string(ftr->type));
+ error("something: %s\t%"PRIu32"\t%"PRIu32"\t%s\t%s\n", aux->seq[ftr->iseq],ftr->beg+1,ftr->end+1,gff->tscript_ids.str[ftr->trid],gf_type2gff_string(ftr->type));
}
tscript_init_cds(gff);
INC_NWARN(wrong_phase);
INC_NWARN(overlapping_cds);
if ( nwarn > 0 )
- fprintf(bcftools_stderr,"Warning: %d warnings were supressed, run with `--verbose 2` to see them all\n",nwarn);
+ fprintf(bcftools_stderr,"Warning: %d warnings were suppressed, increase verbosity to see them all\n",nwarn);
}
if ( gff->dump_fname ) gff_dump(gff, gff->dump_fname);
/* The MIT License
- Copyright (c) 2023 Genome Research Ltd.
+ Copyright (c) 2023-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
Read about transcript types here
http://vega.sanger.ac.uk/info/about/gene_and_transcript_types.html
- http://www.ensembl.org/info/genome/variation/predicted_data.html
+ https://www.ensembl.org/info/genome/variation/prediction/predicted_data.html
https://www.gencodegenes.org/pages/biotypes.html
List of supported biotypes
#ifndef GFF_H__
#define GFF_H__
-#include <stdio.h>
-#include <stdlib.h>
-#include <assert.h>
-#include <getopt.h>
-#include <math.h>
-#include <inttypes.h>
-#include <htslib/hts.h>
-#include <htslib/khash.h>
-#include <htslib/khash_str2int.h>
-#include <htslib/kseq.h>
-#include <htslib/faidx.h>
-#include <htslib/bgzf.h>
-#include <errno.h>
-#include <unistd.h>
-#include <ctype.h>
-#include "bcftools.h"
-#include "regidx.h"
+#include <stdint.h>
#ifndef __FUNCTION__
# define __FUNCTION__ __func__
#define STRAND_REV 0
#define STRAND_FWD 1
+#define STRAND_UNK 2
#define TRIM_NONE 0
#define TRIM_5PRIME 1
{
uint32_t id; // transcript id
uint32_t beg,end; // transcript's beg and end coordinate (ref strand, 0-based, inclusive)
- uint32_t strand:1, // STRAND_REV or STRAND_FWD
+ uint32_t strand:2, // STRAND_REV,FWD,UNK
used:1, // does it have any exons, UTRs, CDS?
- ncds:30, // number of exons
+ ncds:29, // number of exons
mcds;
gf_cds_t **cds; // ordered list of exons
uint32_t trim:2, // complete, 5' or 3' trimmed, see TRIM_* types
void gvcf_update_header(gvcf_t *gvcf, bcf_hdr_t *hdr)
{
bcf_hdr_append(hdr,"##INFO=<ID=END,Number=1,Type=Integer,Description=\"End position of the variant described in this record\">");
- bcf_hdr_append(hdr,"##INFO=<ID=MinDP,Number=1,Type=Integer,Description=\"Minimum per-sample depth in this gVCF block\">");
+ bcf_hdr_append(hdr,"##INFO=<ID=MIN_DP,Number=1,Type=Integer,Description=\"Minimum per-sample depth in this gVCF block\">");
}
gvcf_t *gvcf_init(const char *dp_ranges)
// encountered, or other conditions not met (block broken by a non-ref or DP too low).
int needs_flush = can_collapse ? 0 : 1;
-
// Can the record be included in a gVCF block? That is, is this a ref-only site?
if ( rec && can_collapse )
{
bcf_update_alleles_str(hdr, gvcf->line, gvcf->als.s);
if ( gvcf->start+1 < gvcf->end ) // create gVCF record only if it spans at least two sites
bcf_update_info_int32(hdr, gvcf->line, "END", &gvcf->end, 1);
- bcf_update_info_int32(hdr, gvcf->line, "MinDP", &gvcf->min_dp, 1);
+ bcf_update_info_int32(hdr, gvcf->line, "MIN_DP", &gvcf->min_dp, 1);
if ( gvcf->nqsum>0 )
bcf_update_info_float(hdr, gvcf->line, "QS", gvcf->qsum, gvcf->nqsum);
if ( gvcf->ngts )
}
if ( is_ref && min_dp )
- bcf_update_info_int32(hdr, rec, "MinDP", &min_dp, 1);
+ bcf_update_info_int32(hdr, rec, "MIN_DP", &min_dp, 1);
return rec;
}
void gvcf_update_header(gvcf_t *gvcf, bcf_hdr_t *hdr)
{
bcf_hdr_append(hdr,"##INFO=<ID=END,Number=1,Type=Integer,Description=\"End position of the variant described in this record\">");
- bcf_hdr_append(hdr,"##INFO=<ID=MinDP,Number=1,Type=Integer,Description=\"Minimum per-sample depth in this gVCF block\">");
+ bcf_hdr_append(hdr,"##INFO=<ID=MIN_DP,Number=1,Type=Integer,Description=\"Minimum per-sample depth in this gVCF block\">");
}
gvcf_t *gvcf_init(const char *dp_ranges)
// encountered, or other conditions not met (block broken by a non-ref or DP too low).
int needs_flush = can_collapse ? 0 : 1;
-
// Can the record be included in a gVCF block? That is, is this a ref-only site?
if ( rec && can_collapse )
{
bcf_update_alleles_str(hdr, gvcf->line, gvcf->als.s);
if ( gvcf->start+1 < gvcf->end ) // create gVCF record only if it spans at least two sites
bcf_update_info_int32(hdr, gvcf->line, "END", &gvcf->end, 1);
- bcf_update_info_int32(hdr, gvcf->line, "MinDP", &gvcf->min_dp, 1);
+ bcf_update_info_int32(hdr, gvcf->line, "MIN_DP", &gvcf->min_dp, 1);
if ( gvcf->nqsum>0 )
bcf_update_info_float(hdr, gvcf->line, "QS", gvcf->qsum, gvcf->nqsum);
if ( gvcf->ngts )
}
if ( is_ref && min_dp )
- bcf_update_info_int32(hdr, rec, "MinDP", &min_dp, 1);
+ bcf_update_info_int32(hdr, rec, "MIN_DP", &min_dp, 1);
return rec;
}
if (argc < 2) { usage(stderr); return 1; }
if (strcmp(argv[1], "version") == 0 || strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-v") == 0) {
- printf("bcftools %s\nUsing htslib %s\nCopyright (C) 2023 Genome Research Ltd.\n", bcftools_version(), hts_version());
+ printf("bcftools %s\nUsing htslib %s\nCopyright (C) 2024 Genome Research Ltd.\n", bcftools_version(), hts_version());
#if USE_GPL
printf("License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\n");
#else
if (argc < 2) { usage(bcftools_stderr); return 1; }
if (strcmp(argv[1], "version") == 0 || strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-v") == 0) {
- fprintf(bcftools_stdout, "bcftools %s\nUsing htslib %s\nCopyright (C) 2023 Genome Research Ltd.\n", bcftools_version(), hts_version());
+ fprintf(bcftools_stdout, "bcftools %s\nUsing htslib %s\nCopyright (C) 2024 Genome Research Ltd.\n", bcftools_version(), hts_version());
#if USE_GPL
fprintf(bcftools_stdout, "License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\n");
#else
// qual calculation is not affected.
// Missing values are replaced by generic likelihoods when X (unseen allele) is
// present.
-// NB: While the -m callig model uses the pdgs in canonical order,
+// NB: While the -m calling model uses the pdgs in canonical order,
// the original samtools -c calling code uses pdgs in reverse order (AA comes
// first, RR last).
// NB: Ploidy is not taken into account here, which is incorrect.
// If available, take into account reference panel AFs
if ( call->prior_AN && bcf_get_info_int32(call->hdr, rec, call->prior_AN ,&call->ac, &call->nac)==1 )
{
- int an = call->ac[0]; // number of alleles total, procede only if not zero; reuse call->ac
+ int an = call->ac[0]; // number of alleles total, proceed only if not zero; reuse call->ac
if ( an > 0 && bcf_get_info_int32(call->hdr, rec, call->prior_AC ,&call->ac, &call->nac)==nals_ori-1 ) // number of ALT alleles
{
int ac0 = an; // this will become the number of REFs
call->nals_new = 0;
for (i=0; i<nals_ori; i++)
{
+ if ( (call->flag&CALL_KEEP_UNSEEN) && i==unseen && call->nals_new==1 )
+ {
+ call->nals_new++;
+ call->als_new |= 1<<i;
+ }
if ( i>0 && i==unseen ) continue;
if ( call->flag & CALL_KEEPALT ) call->als_new |= 1<<i;
if ( call->als_new & (1<<i) ) call->nals_new++;
bcf_update_info_int32(call->hdr, rec, "I16", NULL, 0); // remove I16 tag
- return call->nals_new;
+ return is_variant ? call->nals_new : 1;
}
// qual calculation is not affected.
// Missing values are replaced by generic likelihoods when X (unseen allele) is
// present.
-// NB: While the -m callig model uses the pdgs in canonical order,
+// NB: While the -m calling model uses the pdgs in canonical order,
// the original samtools -c calling code uses pdgs in reverse order (AA comes
// first, RR last).
// NB: Ploidy is not taken into account here, which is incorrect.
// If available, take into account reference panel AFs
if ( call->prior_AN && bcf_get_info_int32(call->hdr, rec, call->prior_AN ,&call->ac, &call->nac)==1 )
{
- int an = call->ac[0]; // number of alleles total, procede only if not zero; reuse call->ac
+ int an = call->ac[0]; // number of alleles total, proceed only if not zero; reuse call->ac
if ( an > 0 && bcf_get_info_int32(call->hdr, rec, call->prior_AC ,&call->ac, &call->nac)==nals_ori-1 ) // number of ALT alleles
{
int ac0 = an; // this will become the number of REFs
call->nals_new = 0;
for (i=0; i<nals_ori; i++)
{
+ if ( (call->flag&CALL_KEEP_UNSEEN) && i==unseen && call->nals_new==1 )
+ {
+ call->nals_new++;
+ call->als_new |= 1<<i;
+ }
if ( i>0 && i==unseen ) continue;
if ( call->flag & CALL_KEEPALT ) call->als_new |= 1<<i;
if ( call->als_new & (1<<i) ) call->nals_new++;
bcf_update_info_int32(call->hdr, rec, "I16", NULL, 0); // remove I16 tag
- return call->nals_new;
+ return is_variant ? call->nals_new : 1;
}
/* mpileup.c -- mpileup subcommand. Previously bam_plcmd.c from samtools
- Copyright (C) 2008-2023 Genome Research Ltd.
+ Copyright (C) 2008-2024 Genome Research Ltd.
Portions copyright (C) 2009-2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
uint32_t fmt_flag;
int rflag_skip_any_unset, rflag_skip_all_unset, rflag_skip_any_set, rflag_skip_all_set, output_type;
int openQ, extQ, tandemQ, min_support, indel_win_size; // for indels
+ int seqQ_offset;
double min_frac; // for indels
- double indel_bias;
+ double indel_bias, poly_mqual;
+ double del_bias; // compensate for diff deletion vs insertion error rates
+ double vs_ref;
char *reg_fname, *pl_list, *fai_fname, *output_fname;
int reg_is_file, record_cmd_line, n_threads, clevel;
faidx_t *fai;
htsFile *bcf_fp;
bcf_hdr_t *bcf_hdr;
int indels_v20;
+ int edlib;
int argc;
char **argv;
int write_index;
// call indels; todo: subsampling with total_depth>max_indel_depth instead of ignoring?
// check me: rghash in bcf_call_gap_prep() should have no effect, reads mplp_func already excludes them
- if ( !(conf->flag&MPLP_NO_INDEL) && total_depth < conf->max_indel_depth )
+ if (!(conf->flag&MPLP_NO_INDEL) && total_depth < conf->max_indel_depth)
{
bcf_callaux_clean(conf->bca, &conf->bc);
conf->bca->chr = tid>=0 ? hdr->target_name[tid] : NULL;
int iret;
- if ( conf->indels_v20 )
+ if (conf->edlib)
+ iret = bcf_edlib_gap_prep(conf->gplp->n, conf->gplp->n_plp, conf->gplp->plp, pos, conf->bca, ref, ref_len);
+ else if ( conf->indels_v20 )
iret = bcf_iaux_gap_prep(conf->gplp->n, conf->gplp->n_plp, conf->gplp->plp, pos, conf->bca, ref);
else
iret = bcf_call_gap_prep(conf->gplp->n, conf->gplp->n_plp, conf->gplp->plp, pos, conf->bca, ref);
}
}
}
- return 0;
+ return ret;
}
static int mpileup(mplp_conf_t *conf)
}
}
nregs = regidx_nregs(conf->reg);
- conf->reg_itr = regitr_init(conf->reg);
- regitr_loop(conf->reg_itr); // region iterator now positioned at the first region
+ if ( nregs )
+ {
+ // the regions list can be empty, see #2250
+ conf->reg_itr = regitr_init(conf->reg);
+ regitr_loop(conf->reg_itr); // region iterator now positioned at the first region
+ }
}
// read the header of each file in the list and initialize data
i--;
continue;
}
- if (conf->reg) {
+ if (conf->reg && nregs) {
hts_idx_t *idx = sam_index_load(conf->mplp_data[i]->fp, conf->files[i]);
if (idx == NULL) {
fprintf(stderr, "[%s] fail to load index for %s\n", __func__, conf->files[i]);
for (i=0; i<nsmpl; i++)
bcf_hdr_add_sample(conf->bcf_hdr, smpl[i]);
if ( bcf_hdr_write(conf->bcf_fp, conf->bcf_hdr)!=0 ) error("[%s] Error: failed to write the header to %s\n",__func__,conf->output_fname?conf->output_fname:"standard output");
- if ( conf->write_index && init_index(conf->bcf_fp,conf->bcf_hdr,conf->output_fname,&conf->index_fn)<0 ) error("Error: failed to initialise index for %s\n",conf->output_fname);
+ if ( init_index2(conf->bcf_fp,conf->bcf_hdr,conf->output_fname,
+ &conf->index_fn, conf->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",conf->output_fname);
conf->bca = bcf_call_init(-1., conf->min_baseQ, conf->max_baseQ,
conf->delta_baseQ);
conf->bcr = (bcf_callret1_t*) calloc(nsmpl, sizeof(bcf_callret1_t));
conf->bca->openQ = conf->openQ, conf->bca->extQ = conf->extQ, conf->bca->tandemQ = conf->tandemQ;
conf->bca->indel_bias = conf->indel_bias;
+ conf->bca->del_bias = conf->del_bias;
conf->bca->min_frac = conf->min_frac;
conf->bca->min_support = conf->min_support;
conf->bca->per_sample_flt = conf->flag & MPLP_PER_SAMPLE;
conf->bca->ambig_reads = conf->ambig_reads;
conf->bca->indel_win_size = conf->indel_win_size;
conf->bca->indels_v20 = conf->indels_v20;
+ conf->bca->edlib = conf->edlib;
+ conf->bca->seqQ_offset = conf->seqQ_offset;
+ conf->bca->poly_mqual = conf->poly_mqual;
+ conf->bca->vs_ref = conf->vs_ref;
conf->bc.bcf_hdr = conf->bcf_hdr;
conf->bc.n = nsmpl;
// Run mpileup for multiple regions
+ int ret = 0;
if ( nregs )
{
int ireg = 0;
bam_mplp_reset(conf->iter);
}
}
- mpileup_reg(conf,conf->reg_itr->beg,conf->reg_itr->end);
+ ret = mpileup_reg(conf,conf->reg_itr->beg,conf->reg_itr->end);
+ if ( ret<0 ) break;
}
while ( regitr_loop(conf->reg_itr) );
}
- else
- mpileup_reg(conf,0,UINT32_MAX);
+ else if ( !conf->reg )
+ ret = mpileup_reg(conf,0,UINT32_MAX);
+ if ( ret<0 )
+ {
+ fprintf(stderr, "[%s] failed to read from input file\n", __func__);
+ exit(EXIT_FAILURE);
+ }
flush_bcf_records(conf, conf->bcf_fp, conf->bcf_hdr, NULL);
"\n"
"FORMAT annotation tags available (\"FORMAT/\" prefix is optional):\n"
"\n"
- " FORMAT/AD .. Allelic depth (Number=R,Type=Integer)\n"
+ "* FORMAT/AD .. Allelic depth (Number=R,Type=Integer)\n"
" FORMAT/ADF .. Allelic depths on the forward strand (Number=R,Type=Integer)\n"
" FORMAT/ADR .. Allelic depths on the reverse strand (Number=R,Type=Integer)\n"
" FORMAT/DP .. Number of high-quality bases (Number=1,Type=Integer)\n"
"\n"
"Input options:\n"
" -6, --illumina1.3+ Quality is in the Illumina-1.3+ encoding\n"
- " -A, --count-orphans Do not discard anomalous read pairs\n"
+ " -A, --count-orphans Include anomalous read pairs, with flag PAIRED but not PROPER_PAIR set\n"
" -b, --bam-list FILE List of input BAM filenames, one per line\n"
" -B, --no-BAQ Disable BAQ (per-Base Alignment Quality)\n"
" -C, --adjust-MQ INT Adjust mapping quality [0]\n"
" -O, --output-type TYPE 'b' compressed BCF; 'u' uncompressed BCF;\n"
" 'z' compressed VCF; 'v' uncompressed VCF; 0-9 compression level [v]\n"
" --threads INT Use multithreading with INT worker threads [0]\n"
- " --write-index Automatically index the output files [off]\n"
+ " -W, --write-index[=FMT] Automatically index the output files [off]\n"
"\n"
"SNP/INDEL genotype likelihoods options:\n"
- " -X, --config STR Specify platform specific profiles (see below)\n"
+ " -X, --config STR Specify platform profile (use \"-X list\" for details)\n"
" -e, --ext-prob INT Phred-scaled gap extension seq error probability [%d]\n", mplp->extQ);
fprintf(fp,
" -F, --gap-frac FLOAT Minimum fraction of gapped reads [%g]\n", mplp->min_frac);
" --ar, --ambig-reads STR What to do with ambiguous indel reads: drop,incAD,incAD0 [drop]\n");
fprintf(fp,
" --indel-bias FLOAT Raise to favour recall over precision [%.2f]\n", mplp->indel_bias);
+ fprintf(fp,
+ " --del-bias FLOAT Relative likelihood of insertion to deletion [%.2f]\n", mplp->del_bias);
+ fprintf(fp,
+ " --score-vs-ref FLOAT\n"
+ " Ratio of score vs ref (1) or 2nd-best allele (0) [%.2f]\n", mplp->vs_ref);
fprintf(fp,
" --indel-size INT Approximate maximum indel size considered [%d]\n", mplp->indel_win_size);
fprintf(fp,
- " --indels-2.0 New EXPERIMENTAL indel calling model (diploid reference consensus)\n");
+ " --indels-2.0 New EXPERIMENTAL indel calling model (diploid reference consensus)\n"
+ " --indels-cns New EXPERIMENTAL indel calling model with edlib\n"
+ " --seqq-offset Indel-cns tuning for indel seq-qual scores [120]\n"
+ " --no-indels-cns Disable CNS mode, to use after a -X profile\n"
+ " --poly-mqual (Edlib mode) Use minimum quality within homopolymers\n");
fprintf(fp,"\n");
fprintf(fp,
- "Configuration profiles activated with -X, --config:\n"
- " 1.12: -Q13 -h100 -m1 -F0.002\n"
- " illumina: [ default values ]\n"
- " ont: -B -Q5 --max-BQ 30 -I [also try eg |bcftools call -P0.01]\n"
- " pacbio-ccs: -D -Q5 --max-BQ 50 -F0.1 -o25 -e1 --delta-BQ 10 -M99999\n"
- "\n"
- "Notes: Assuming diploid individuals.\n"
- "\n"
- "Example:\n"
- " # See also http://samtools.github.io/bcftools/howtos/variant-calling.html\n"
- " bcftools mpileup -Ou -f reference.fa alignments.bam | bcftools call -mv -Ob -o calls.bcf\n"
- "\n");
+ "Notes: Assuming diploid individuals.\n\n"
+ "Example:\n"
+ " # See also http://samtools.github.io/bcftools/howtos/variant-calling.html\n"
+ " bcftools mpileup -Ou -f reference.fa alignments.bam | bcftools call -mv -Ob -o calls.bcf\n"
+ "\n");
free(tmp_skip_all_set);
free(tmp_skip_any_unset);
free(tmp_skip_any_set);
}
+static void print_profiles(void) {
+ printf(
+"Configuration profiles activated with -X, --config:\n\n"
+"1.12\n"
+" -Q13 -h100 -m1 -F0.002\n\n"
+"bgi, bgi-1.20\n"
+" --indels-cns -B --indel-size 80 -F0.1 --indel-bias 0.9 --seqq-offset 120\n\n"
+"illumina-1.18\n"
+" --indel-size 110\n\n"
+"illumina\n"
+"illumina-1.20\n"
+" --indels-cns --indel-size 110\n\n"
+"ont\n"
+" -B -Q5 --max-BQ 30 -I\n\n"
+"ont-sup, ont-sup-1.20\n"
+" --indels-cns -B -Q1 --max-BQ 35 -F0.2 -o15 -e1 -h110 --delta-BQ 99\\\n"
+" --del-bias 0.4 --indel-bias 0.7 --poly-mqual --seqq-offset 130\\\n"
+" --indel-size 80\n\n"
+"pacbio-ccs-1.18\n"
+" -D -Q5 --max-BQ 50 -F0.1 -o25 -e1 --delta-BQ 10 \\\n"
+" -M99999 --indel-size 110\n\n"
+"pacbio-ccs, pacbio-ccs-1.20\n"
+" --indels-cns -B -Q5 --max-BQ 50 -F0.1 -o25 -e1 -h300 --delta-BQ 10 \\\n"
+" --del-bias 0.4 --poly-mqual --indel-bias 0.9 --seqq-offset 118\\\n"
+" --indel-size 80 --score-vs-ref 0.7\n\n"
+"ultima, ultima-1.20\n"
+" --indels-cns -B -Q1 --max-BQ 30 -F0.15 -o20 -e10 -h250 --delta-BQ 10 \\\n"
+" --del-bias 0.3 --indel-bias 0.7 --poly-mqual --seqq-offset 140 \\\n"
+" --indel-size 80 --score-vs-ref 0.3\n\n"
+"\n");
+}
+
int main_mpileup(int argc, char *argv[])
{
- int c;
+ int c, i, ret = 1;
const char *file_list = NULL;
char **fn = NULL;
int nfiles = 0, use_orphan = 0, noref = 0;
mplp.max_depth = 250; mplp.max_indel_depth = 250;
mplp.openQ = 40; mplp.extQ = 20; mplp.tandemQ = 500;
mplp.min_frac = 0.05; mplp.indel_bias = 1.0; mplp.min_support = 2;
+ mplp.vs_ref = 0;
mplp.flag = MPLP_NO_ORPHAN | MPLP_REALN | MPLP_REALN_PARTIAL
| MPLP_SMART_OVERLAPS;
mplp.argc = argc; mplp.argv = argv;
mplp.n_threads = 0;
mplp.bsmpl = bam_smpl_init();
// the default to be changed in future, see also parse_format_flag()
- mplp.fmt_flag = B2B_INFO_BQBZ|B2B_INFO_IDV|B2B_INFO_IMF|B2B_INFO_MQ0F|B2B_INFO_MQBZ|B2B_INFO_MQSBZ|B2B_INFO_RPBZ|B2B_INFO_SCBZ|B2B_INFO_SGB|B2B_INFO_VDB;
+ mplp.fmt_flag = B2B_INFO_BQBZ|B2B_INFO_IDV|B2B_INFO_IMF|B2B_INFO_MQ0F|B2B_INFO_MQBZ|B2B_INFO_MQSBZ|B2B_INFO_RPBZ|B2B_INFO_SCBZ|B2B_INFO_SGB|B2B_INFO_VDB|B2B_FMT_AD;
mplp.max_read_len = 500;
mplp.ambig_reads = B2B_DROP;
mplp.indel_win_size = 110;
+ mplp.poly_mqual = 0;
+ mplp.seqQ_offset = 120;
mplp.clevel = -1;
+ mplp.del_bias = 0; // even insertion and deletion likelhoods.
hts_srand48(0);
static const struct option lopts[] =
{"indel-bias", required_argument, NULL, 10},
{"indel-size", required_argument, NULL, 15},
{"indels-2.0", no_argument, NULL, 20},
+ {"indels-cns", no_argument, NULL, 22},
+ {"no-indels-cns", no_argument, NULL, 25},
{"tandem-qual", required_argument, NULL, 'h'},
{"skip-indels", no_argument, NULL, 'I'},
{"max-idepth", required_argument, NULL, 'L'},
{"seed", required_argument, NULL, 13},
{"ambig-reads", required_argument, NULL, 14},
{"ar", required_argument, NULL, 14},
- {"write-index",no_argument,NULL,21},
+ {"write-index",optional_argument,NULL,'W'},
+ {"del-bias", required_argument, NULL, 23},
+ {"poly-mqual", no_argument, NULL, 24},
+ {"no-poly-mqual", no_argument, NULL, 26},
+ {"score-vs-ref",required_argument, NULL, 27},
+ {"seqq-offset", required_argument, NULL, 28},
{NULL, 0, NULL, 0}
};
- while ((c = getopt_long(argc, argv, "Ag:f:r:R:q:Q:C:BDd:L:b:P:po:e:h:Im:F:EG:6O:xa:s:S:t:T:M:X:U",lopts,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "Ag:f:r:R:q:Q:C:BDd:L:b:P:po:e:h:Im:F:EG:6O:xa:s:S:t:T:M:X:UW::",lopts,NULL)) >= 0) {
switch (c) {
case 'x': mplp.flag &= ~MPLP_SMART_OVERLAPS; break;
case 16 :
mplp.rflag_skip_any_unset = bam_str2flag(optarg);
- if ( mplp.rflag_skip_any_unset <0 ) { fprintf(stderr,"Could not parse --nf %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_any_unset <0 ) {
+ fprintf(stderr,"Could not parse --nf %s\n", optarg);
+ goto err;
+ }
break;
case 17 :
mplp.rflag_skip_all_unset = bam_str2flag(optarg);
- if ( mplp.rflag_skip_all_unset<0 ) { fprintf(stderr,"Could not parse --if %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_all_unset<0 ) {
+ fprintf(stderr,"Could not parse --if %s\n", optarg);
+ goto err;
+ }
break;
case 18 :
mplp.rflag_skip_any_set = bam_str2flag(optarg);
- if ( mplp.rflag_skip_any_set <0 ) { fprintf(stderr,"Could not parse --ef %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_any_set <0 ) {
+ fprintf(stderr,"Could not parse --ef %s\n", optarg);
+ goto err;
+ }
break;
case 19 :
mplp.rflag_skip_all_set = bam_str2flag(optarg);
- if ( mplp.rflag_skip_all_set <0 ) { fprintf(stderr,"Could not parse --df %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_all_set <0 ) {
+ fprintf(stderr,"Could not parse --df %s\n", optarg);
+ goto err;
+ }
break;
case 3 : mplp.output_fname = optarg; break;
case 4 : mplp.openQ = atoi(optarg); break;
break;
case 'f':
mplp.fai = fai_load(optarg);
- if (mplp.fai == NULL) return 1;
+ if (mplp.fai == NULL)
+ goto err;
mplp.fai_fname = optarg;
break;
case 7 : noref = 1; break;
if ( optarg[0]=='^' ) optarg++;
else mplp.bed_logic = 1;
mplp.bed = regidx_init(optarg,NULL,NULL,0,NULL);
- if (!mplp.bed) { fprintf(stderr, "bcftools mpileup: Could not read file \"%s\"", optarg); return 1; }
+ if (!mplp.bed) {
+ fprintf(stderr, "bcftools mpileup: Could not read file \"%s\"", optarg);
+ goto err;
+ }
break;
case 'P': mplp.pl_list = strdup(optarg); break;
case 'p': mplp.flag |= MPLP_PER_SAMPLE; break;
else
mplp.indel_bias = 1/atof(optarg);
break;
+ case 27:
+ mplp.vs_ref = atof(optarg);
+ //if (mplp.vs_ref < 0) mplp.vs_ref = 0;
+ if (mplp.vs_ref > 1) mplp.vs_ref = 1;
+ break;
case 15: {
char *tmp;
mplp.indel_win_size = strtol(optarg,&tmp,10);
if ( *tmp ) error("Could not parse argument: --indel-size %s\n", optarg);
- if ( mplp.indel_win_size < 110 )
+ if ( mplp.indel_win_size < 20 )
{
- mplp.indel_win_size = 110;
+ mplp.indel_win_size = 20;
fprintf(stderr,"Warning: running with --indel-size %d, the requested value is too small\n",mplp.indel_win_size);
}
}
break;
- case 20: mplp.indels_v20 = 1; break;
- case 21: mplp.write_index = 1; break;
+ case 20: mplp.indels_v20 = 1; mplp.edlib = 0; break;
+ case 'W':
+ if (!(mplp.write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
+ case 22: mplp.edlib = 1; mplp.indels_v20 = 0; break;
+ case 25: mplp.edlib = 0; break;
+ case 28:
+ mplp.seqQ_offset = atoi(optarg);
+ if (mplp.seqQ_offset < 100)
+ mplp.seqQ_offset = 100;
+ if (mplp.seqQ_offset > 200)
+ mplp.seqQ_offset = 200;
+ break;
+ case 23: mplp.del_bias = atof(optarg); break;
+ case 24: mplp.poly_mqual = 1; break;
+ case 26: mplp.poly_mqual = 0; break;
case 'A': use_orphan = 1; break;
case 'F': mplp.min_frac = atof(optarg); break;
case 'm': mplp.min_support = atoi(optarg); break;
case 'a':
if (optarg[0]=='?') {
list_annotations(stderr);
- return 1;
+ goto err;
}
parse_format_flag(&mplp.fmt_flag,optarg);
break;
case 'M': mplp.max_read_len = atoi(optarg); break;
case 'X':
- if (strcasecmp(optarg, "pacbio-ccs") == 0) {
+ if (strcasecmp(optarg, "pacbio-ccs-1.18") == 0) {
mplp.min_frac = 0.1;
mplp.min_baseQ = 5;
mplp.max_baseQ = 50;
mplp.extQ = 1;
mplp.flag |= MPLP_REALN_PARTIAL;
mplp.max_read_len = 99999;
+
+ } else if (strcasecmp(optarg, "pacbio-ccs") == 0 ||
+ strcasecmp(optarg, "pacbio-ccs-1.20") == 0) {
+ mplp.min_frac = 0.1;
+ mplp.min_baseQ = 5;
+ mplp.max_baseQ = 50;
+ mplp.delta_baseQ = 10;
+ mplp.tandemQ = 300;
+ mplp.openQ = 25;
+ mplp.extQ = 1;
+ mplp.flag &= ~MPLP_REALN;
+ mplp.del_bias = 0.4;
+ mplp.indel_bias = 1/.9;
+ mplp.seqQ_offset = 118;
+ mplp.poly_mqual = 1;
+ mplp.edlib = 1;
+ mplp.vs_ref = 0.7;
+ mplp.indel_win_size = 80;
+
} else if (strcasecmp(optarg, "ont") == 0) {
- fprintf(stderr, "For ONT it may be beneficial to also run bcftools call with "
+ fprintf(stderr, "With old ONT data may be beneficial to also run bcftools call with "
"a higher -P, eg -P0.01 or -P 0.1\n");
mplp.min_baseQ = 5;
mplp.max_baseQ = 30;
mplp.flag &= ~MPLP_REALN;
mplp.flag |= MPLP_NO_INDEL;
+
+ } else if (strcasecmp(optarg, "ont-sup") == 0 ||
+ strcasecmp(optarg, "ont-sup-1.20") == 0) {
+ mplp.min_frac = 0.2;
+ mplp.min_baseQ = 1;
+ mplp.max_baseQ = 35;
+ mplp.delta_baseQ = 99;
+ mplp.openQ = 15;
+ mplp.extQ = 1;
+ mplp.flag &= ~MPLP_REALN;
+ mplp.max_read_len = 9999999;
+ mplp.del_bias = 0.4;
+ mplp.poly_mqual = 1;
+ mplp.edlib = 1;
+ // If we increase -h then we can increase bias denominator too
+ mplp.tandemQ = 110;
+ mplp.indel_bias = 1/0.7;
+ mplp.seqQ_offset = 130;
+ mplp.indel_win_size = 80;
+
+ } else if (strcasecmp(optarg, "ultima") == 0 ||
+ strcasecmp(optarg, "ultima-1.20") == 0) {
+ mplp.min_frac = 0.15;
+ mplp.min_baseQ = 1;
+ mplp.max_baseQ = 30;
+ mplp.delta_baseQ = 10;
+ mplp.openQ = 20;
+ mplp.extQ = 10;
+ mplp.tandemQ = 250;
+ mplp.flag &= ~MPLP_REALN;
+ mplp.del_bias = 0.3;
+ mplp.poly_mqual = 1;
+ mplp.edlib = 1;
+ mplp.indel_bias = 1/0.7;
+ mplp.seqQ_offset = 140;
+ mplp.vs_ref = 0.3;
+ mplp.indel_win_size = 80;
+
} else if (strcasecmp(optarg, "1.12") == 0) {
// 1.12 and earlier
mplp.min_frac = 0.002;
mplp.tandemQ = 100;
mplp.flag &= ~MPLP_REALN_PARTIAL;
mplp.flag |= MPLP_REALN;
- } else if (strcasecmp(optarg, "illumina") == 0) {
+
+ } else if (strcasecmp(optarg, "illumina-1.18") == 0) {
+ mplp.indel_win_size = 110;
+ mplp.flag |= MPLP_REALN_PARTIAL;
+
+ } else if (strcasecmp(optarg, "illumina") == 0 ||
+ strcasecmp(optarg, "illumina-1.20") == 0) {
+ mplp.edlib = 1;
+ mplp.indel_win_size = 110;
mplp.flag |= MPLP_REALN_PARTIAL;
+ mplp.indel_bias = 1;
+ mplp.seqQ_offset = 125;
+ //mplp.indel_win_size = 80; TEST?
+
+ } else if (strcasecmp(optarg, "bgi") == 0 ||
+ strcasecmp(optarg, "bgi-1.20") == 0) {
+ mplp.min_frac = 0.1;
+ mplp.edlib = 1;
+ mplp.indel_bias = 1;
+ mplp.seqQ_offset = 120;
+ mplp.flag |= MPLP_REALN_PARTIAL;
+ mplp.indel_win_size = 80;
+
+ } else if (strcasecmp(optarg, "list") == 0 ||
+ strcasecmp(optarg, "help") == 0) {
+ print_profiles();
+ goto err;
} else {
fprintf(stderr, "Unknown configuration name '%s'\n"
- "Please choose from 1.12, illumina, pacbio-ccs or ont\n",
+ "Please use '-X list' to show available choices.\n",
optarg);
- return 1;
+ goto err;
}
break;
case 13: hts_srand48(atoi(optarg)); break;
break;
default:
fprintf(stderr,"Invalid option: '%c'\n", c);
- return 1;
+ goto err;
}
}
if ( !(mplp.flag&MPLP_REALN) && mplp.flag&MPLP_REDO_BAQ )
{
fprintf(stderr,"Error: The -B option cannot be combined with -E\n");
- return 1;
+ goto err;
}
if (use_orphan) mplp.flag &= ~MPLP_NO_ORPHAN;
if (argc == 1)
{
print_usage(stderr, &mplp);
- return 1;
+ goto err;
}
if (!mplp.fai && !noref) {
fprintf(stderr,"Error: mpileup requires the --fasta-ref option by default; use --no-reference to run without a fasta reference\n");
- return 1;
+ goto err;
}
- int ret,i;
+
if (file_list)
{
- if ( read_file_list(file_list,&nfiles,&fn) ) return 1;
+ if ( read_file_list(file_list,&nfiles,&fn) )
+ goto err;
mplp.files = fn;
mplp.nfiles = nfiles;
}
if (mplp.bed) regidx_destroy(mplp.bed);
if (mplp.bed_itr) regitr_destroy(mplp.bed_itr);
if (mplp.reg) regidx_destroy(mplp.reg);
+
+ err:
bam_smpl_destroy(mplp.bsmpl);
return ret;
/* mpileup.c -- mpileup subcommand. Previously bam_plcmd.c from samtools
- Copyright (C) 2008-2023 Genome Research Ltd.
+ Copyright (C) 2008-2024 Genome Research Ltd.
Portions copyright (C) 2009-2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
uint32_t fmt_flag;
int rflag_skip_any_unset, rflag_skip_all_unset, rflag_skip_any_set, rflag_skip_all_set, output_type;
int openQ, extQ, tandemQ, min_support, indel_win_size; // for indels
+ int seqQ_offset;
double min_frac; // for indels
- double indel_bias;
+ double indel_bias, poly_mqual;
+ double del_bias; // compensate for diff deletion vs insertion error rates
+ double vs_ref;
char *reg_fname, *pl_list, *fai_fname, *output_fname;
int reg_is_file, record_cmd_line, n_threads, clevel;
faidx_t *fai;
htsFile *bcf_fp;
bcf_hdr_t *bcf_hdr;
int indels_v20;
+ int edlib;
int argc;
char **argv;
int write_index;
// call indels; todo: subsampling with total_depth>max_indel_depth instead of ignoring?
// check me: rghash in bcf_call_gap_prep() should have no effect, reads mplp_func already excludes them
- if ( !(conf->flag&MPLP_NO_INDEL) && total_depth < conf->max_indel_depth )
+ if (!(conf->flag&MPLP_NO_INDEL) && total_depth < conf->max_indel_depth)
{
bcf_callaux_clean(conf->bca, &conf->bc);
conf->bca->chr = tid>=0 ? hdr->target_name[tid] : NULL;
int iret;
- if ( conf->indels_v20 )
+ if (conf->edlib)
+ iret = bcf_edlib_gap_prep(conf->gplp->n, conf->gplp->n_plp, conf->gplp->plp, pos, conf->bca, ref, ref_len);
+ else if ( conf->indels_v20 )
iret = bcf_iaux_gap_prep(conf->gplp->n, conf->gplp->n_plp, conf->gplp->plp, pos, conf->bca, ref);
else
iret = bcf_call_gap_prep(conf->gplp->n, conf->gplp->n_plp, conf->gplp->plp, pos, conf->bca, ref);
}
}
}
- return 0;
+ return ret;
}
static int mpileup(mplp_conf_t *conf)
}
}
nregs = regidx_nregs(conf->reg);
- conf->reg_itr = regitr_init(conf->reg);
- regitr_loop(conf->reg_itr); // region iterator now positioned at the first region
+ if ( nregs )
+ {
+ // the regions list can be empty, see #2250
+ conf->reg_itr = regitr_init(conf->reg);
+ regitr_loop(conf->reg_itr); // region iterator now positioned at the first region
+ }
}
// read the header of each file in the list and initialize data
i--;
continue;
}
- if (conf->reg) {
+ if (conf->reg && nregs) {
hts_idx_t *idx = sam_index_load(conf->mplp_data[i]->fp, conf->files[i]);
if (idx == NULL) {
fprintf(bcftools_stderr, "[%s] fail to load index for %s\n", __func__, conf->files[i]);
for (i=0; i<nsmpl; i++)
bcf_hdr_add_sample(conf->bcf_hdr, smpl[i]);
if ( bcf_hdr_write(conf->bcf_fp, conf->bcf_hdr)!=0 ) error("[%s] Error: failed to write the header to %s\n",__func__,conf->output_fname?conf->output_fname:"standard output");
- if ( conf->write_index && init_index(conf->bcf_fp,conf->bcf_hdr,conf->output_fname,&conf->index_fn)<0 ) error("Error: failed to initialise index for %s\n",conf->output_fname);
+ if ( init_index2(conf->bcf_fp,conf->bcf_hdr,conf->output_fname,
+ &conf->index_fn, conf->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",conf->output_fname);
conf->bca = bcf_call_init(-1., conf->min_baseQ, conf->max_baseQ,
conf->delta_baseQ);
conf->bcr = (bcf_callret1_t*) calloc(nsmpl, sizeof(bcf_callret1_t));
conf->bca->openQ = conf->openQ, conf->bca->extQ = conf->extQ, conf->bca->tandemQ = conf->tandemQ;
conf->bca->indel_bias = conf->indel_bias;
+ conf->bca->del_bias = conf->del_bias;
conf->bca->min_frac = conf->min_frac;
conf->bca->min_support = conf->min_support;
conf->bca->per_sample_flt = conf->flag & MPLP_PER_SAMPLE;
conf->bca->ambig_reads = conf->ambig_reads;
conf->bca->indel_win_size = conf->indel_win_size;
conf->bca->indels_v20 = conf->indels_v20;
+ conf->bca->edlib = conf->edlib;
+ conf->bca->seqQ_offset = conf->seqQ_offset;
+ conf->bca->poly_mqual = conf->poly_mqual;
+ conf->bca->vs_ref = conf->vs_ref;
conf->bc.bcf_hdr = conf->bcf_hdr;
conf->bc.n = nsmpl;
// Run mpileup for multiple regions
+ int ret = 0;
if ( nregs )
{
int ireg = 0;
bam_mplp_reset(conf->iter);
}
}
- mpileup_reg(conf,conf->reg_itr->beg,conf->reg_itr->end);
+ ret = mpileup_reg(conf,conf->reg_itr->beg,conf->reg_itr->end);
+ if ( ret<0 ) break;
}
while ( regitr_loop(conf->reg_itr) );
}
- else
- mpileup_reg(conf,0,UINT32_MAX);
+ else if ( !conf->reg )
+ ret = mpileup_reg(conf,0,UINT32_MAX);
+ if ( ret<0 )
+ {
+ fprintf(bcftools_stderr, "[%s] failed to read from input file\n", __func__);
+ bcftools_exit(EXIT_FAILURE);
+ }
flush_bcf_records(conf, conf->bcf_fp, conf->bcf_hdr, NULL);
"\n"
"FORMAT annotation tags available (\"FORMAT/\" prefix is optional):\n"
"\n"
- " FORMAT/AD .. Allelic depth (Number=R,Type=Integer)\n"
+ "* FORMAT/AD .. Allelic depth (Number=R,Type=Integer)\n"
" FORMAT/ADF .. Allelic depths on the forward strand (Number=R,Type=Integer)\n"
" FORMAT/ADR .. Allelic depths on the reverse strand (Number=R,Type=Integer)\n"
" FORMAT/DP .. Number of high-quality bases (Number=1,Type=Integer)\n"
"\n"
"Input options:\n"
" -6, --illumina1.3+ Quality is in the Illumina-1.3+ encoding\n"
- " -A, --count-orphans Do not discard anomalous read pairs\n"
+ " -A, --count-orphans Include anomalous read pairs, with flag PAIRED but not PROPER_PAIR set\n"
" -b, --bam-list FILE List of input BAM filenames, one per line\n"
" -B, --no-BAQ Disable BAQ (per-Base Alignment Quality)\n"
" -C, --adjust-MQ INT Adjust mapping quality [0]\n"
" -O, --output-type TYPE 'b' compressed BCF; 'u' uncompressed BCF;\n"
" 'z' compressed VCF; 'v' uncompressed VCF; 0-9 compression level [v]\n"
" --threads INT Use multithreading with INT worker threads [0]\n"
- " --write-index Automatically index the output files [off]\n"
+ " -W, --write-index[=FMT] Automatically index the output files [off]\n"
"\n"
"SNP/INDEL genotype likelihoods options:\n"
- " -X, --config STR Specify platform specific profiles (see below)\n"
+ " -X, --config STR Specify platform profile (use \"-X list\" for details)\n"
" -e, --ext-prob INT Phred-scaled gap extension seq error probability [%d]\n", mplp->extQ);
fprintf(fp,
" -F, --gap-frac FLOAT Minimum fraction of gapped reads [%g]\n", mplp->min_frac);
" --ar, --ambig-reads STR What to do with ambiguous indel reads: drop,incAD,incAD0 [drop]\n");
fprintf(fp,
" --indel-bias FLOAT Raise to favour recall over precision [%.2f]\n", mplp->indel_bias);
+ fprintf(fp,
+ " --del-bias FLOAT Relative likelihood of insertion to deletion [%.2f]\n", mplp->del_bias);
+ fprintf(fp,
+ " --score-vs-ref FLOAT\n"
+ " Ratio of score vs ref (1) or 2nd-best allele (0) [%.2f]\n", mplp->vs_ref);
fprintf(fp,
" --indel-size INT Approximate maximum indel size considered [%d]\n", mplp->indel_win_size);
fprintf(fp,
- " --indels-2.0 New EXPERIMENTAL indel calling model (diploid reference consensus)\n");
+ " --indels-2.0 New EXPERIMENTAL indel calling model (diploid reference consensus)\n"
+ " --indels-cns New EXPERIMENTAL indel calling model with edlib\n"
+ " --seqq-offset Indel-cns tuning for indel seq-qual scores [120]\n"
+ " --no-indels-cns Disable CNS mode, to use after a -X profile\n"
+ " --poly-mqual (Edlib mode) Use minimum quality within homopolymers\n");
fprintf(fp,"\n");
fprintf(fp,
- "Configuration profiles activated with -X, --config:\n"
- " 1.12: -Q13 -h100 -m1 -F0.002\n"
- " illumina: [ default values ]\n"
- " ont: -B -Q5 --max-BQ 30 -I [also try eg |bcftools call -P0.01]\n"
- " pacbio-ccs: -D -Q5 --max-BQ 50 -F0.1 -o25 -e1 --delta-BQ 10 -M99999\n"
- "\n"
- "Notes: Assuming diploid individuals.\n"
- "\n"
- "Example:\n"
- " # See also http://samtools.github.io/bcftools/howtos/variant-calling.html\n"
- " bcftools mpileup -Ou -f reference.fa alignments.bam | bcftools call -mv -Ob -o calls.bcf\n"
- "\n");
+ "Notes: Assuming diploid individuals.\n\n"
+ "Example:\n"
+ " # See also http://samtools.github.io/bcftools/howtos/variant-calling.html\n"
+ " bcftools mpileup -Ou -f reference.fa alignments.bam | bcftools call -mv -Ob -o calls.bcf\n"
+ "\n");
free(tmp_skip_all_set);
free(tmp_skip_any_unset);
free(tmp_skip_any_set);
}
+static void print_profiles(void) {
+ fprintf(bcftools_stdout,
+"Configuration profiles activated with -X, --config:\n\n"
+"1.12\n"
+" -Q13 -h100 -m1 -F0.002\n\n"
+"bgi, bgi-1.20\n"
+" --indels-cns -B --indel-size 80 -F0.1 --indel-bias 0.9 --seqq-offset 120\n\n"
+"illumina-1.18\n"
+" --indel-size 110\n\n"
+"illumina\n"
+"illumina-1.20\n"
+" --indels-cns --indel-size 110\n\n"
+"ont\n"
+" -B -Q5 --max-BQ 30 -I\n\n"
+"ont-sup, ont-sup-1.20\n"
+" --indels-cns -B -Q1 --max-BQ 35 -F0.2 -o15 -e1 -h110 --delta-BQ 99\\\n"
+" --del-bias 0.4 --indel-bias 0.7 --poly-mqual --seqq-offset 130\\\n"
+" --indel-size 80\n\n"
+"pacbio-ccs-1.18\n"
+" -D -Q5 --max-BQ 50 -F0.1 -o25 -e1 --delta-BQ 10 \\\n"
+" -M99999 --indel-size 110\n\n"
+"pacbio-ccs, pacbio-ccs-1.20\n"
+" --indels-cns -B -Q5 --max-BQ 50 -F0.1 -o25 -e1 -h300 --delta-BQ 10 \\\n"
+" --del-bias 0.4 --poly-mqual --indel-bias 0.9 --seqq-offset 118\\\n"
+" --indel-size 80 --score-vs-ref 0.7\n\n"
+"ultima, ultima-1.20\n"
+" --indels-cns -B -Q1 --max-BQ 30 -F0.15 -o20 -e10 -h250 --delta-BQ 10 \\\n"
+" --del-bias 0.3 --indel-bias 0.7 --poly-mqual --seqq-offset 140 \\\n"
+" --indel-size 80 --score-vs-ref 0.3\n\n"
+"\n");
+}
+
int main_mpileup(int argc, char *argv[])
{
- int c;
+ int c, i, ret = 1;
const char *file_list = NULL;
char **fn = NULL;
int nfiles = 0, use_orphan = 0, noref = 0;
mplp.max_depth = 250; mplp.max_indel_depth = 250;
mplp.openQ = 40; mplp.extQ = 20; mplp.tandemQ = 500;
mplp.min_frac = 0.05; mplp.indel_bias = 1.0; mplp.min_support = 2;
+ mplp.vs_ref = 0;
mplp.flag = MPLP_NO_ORPHAN | MPLP_REALN | MPLP_REALN_PARTIAL
| MPLP_SMART_OVERLAPS;
mplp.argc = argc; mplp.argv = argv;
mplp.n_threads = 0;
mplp.bsmpl = bam_smpl_init();
// the default to be changed in future, see also parse_format_flag()
- mplp.fmt_flag = B2B_INFO_BQBZ|B2B_INFO_IDV|B2B_INFO_IMF|B2B_INFO_MQ0F|B2B_INFO_MQBZ|B2B_INFO_MQSBZ|B2B_INFO_RPBZ|B2B_INFO_SCBZ|B2B_INFO_SGB|B2B_INFO_VDB;
+ mplp.fmt_flag = B2B_INFO_BQBZ|B2B_INFO_IDV|B2B_INFO_IMF|B2B_INFO_MQ0F|B2B_INFO_MQBZ|B2B_INFO_MQSBZ|B2B_INFO_RPBZ|B2B_INFO_SCBZ|B2B_INFO_SGB|B2B_INFO_VDB|B2B_FMT_AD;
mplp.max_read_len = 500;
mplp.ambig_reads = B2B_DROP;
mplp.indel_win_size = 110;
+ mplp.poly_mqual = 0;
+ mplp.seqQ_offset = 120;
mplp.clevel = -1;
+ mplp.del_bias = 0; // even insertion and deletion likelhoods.
hts_srand48(0);
static const struct option lopts[] =
{"indel-bias", required_argument, NULL, 10},
{"indel-size", required_argument, NULL, 15},
{"indels-2.0", no_argument, NULL, 20},
+ {"indels-cns", no_argument, NULL, 22},
+ {"no-indels-cns", no_argument, NULL, 25},
{"tandem-qual", required_argument, NULL, 'h'},
{"skip-indels", no_argument, NULL, 'I'},
{"max-idepth", required_argument, NULL, 'L'},
{"seed", required_argument, NULL, 13},
{"ambig-reads", required_argument, NULL, 14},
{"ar", required_argument, NULL, 14},
- {"write-index",no_argument,NULL,21},
+ {"write-index",optional_argument,NULL,'W'},
+ {"del-bias", required_argument, NULL, 23},
+ {"poly-mqual", no_argument, NULL, 24},
+ {"no-poly-mqual", no_argument, NULL, 26},
+ {"score-vs-ref",required_argument, NULL, 27},
+ {"seqq-offset", required_argument, NULL, 28},
{NULL, 0, NULL, 0}
};
- while ((c = getopt_long(argc, argv, "Ag:f:r:R:q:Q:C:BDd:L:b:P:po:e:h:Im:F:EG:6O:xa:s:S:t:T:M:X:U",lopts,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "Ag:f:r:R:q:Q:C:BDd:L:b:P:po:e:h:Im:F:EG:6O:xa:s:S:t:T:M:X:UW::",lopts,NULL)) >= 0) {
switch (c) {
case 'x': mplp.flag &= ~MPLP_SMART_OVERLAPS; break;
case 16 :
mplp.rflag_skip_any_unset = bam_str2flag(optarg);
- if ( mplp.rflag_skip_any_unset <0 ) { fprintf(bcftools_stderr,"Could not parse --nf %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_any_unset <0 ) {
+ fprintf(bcftools_stderr,"Could not parse --nf %s\n", optarg);
+ goto err;
+ }
break;
case 17 :
mplp.rflag_skip_all_unset = bam_str2flag(optarg);
- if ( mplp.rflag_skip_all_unset<0 ) { fprintf(bcftools_stderr,"Could not parse --if %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_all_unset<0 ) {
+ fprintf(bcftools_stderr,"Could not parse --if %s\n", optarg);
+ goto err;
+ }
break;
case 18 :
mplp.rflag_skip_any_set = bam_str2flag(optarg);
- if ( mplp.rflag_skip_any_set <0 ) { fprintf(bcftools_stderr,"Could not parse --ef %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_any_set <0 ) {
+ fprintf(bcftools_stderr,"Could not parse --ef %s\n", optarg);
+ goto err;
+ }
break;
case 19 :
mplp.rflag_skip_all_set = bam_str2flag(optarg);
- if ( mplp.rflag_skip_all_set <0 ) { fprintf(bcftools_stderr,"Could not parse --df %s\n", optarg); return 1; }
+ if ( mplp.rflag_skip_all_set <0 ) {
+ fprintf(bcftools_stderr,"Could not parse --df %s\n", optarg);
+ goto err;
+ }
break;
case 3 : mplp.output_fname = optarg; break;
case 4 : mplp.openQ = atoi(optarg); break;
break;
case 'f':
mplp.fai = fai_load(optarg);
- if (mplp.fai == NULL) return 1;
+ if (mplp.fai == NULL)
+ goto err;
mplp.fai_fname = optarg;
break;
case 7 : noref = 1; break;
if ( optarg[0]=='^' ) optarg++;
else mplp.bed_logic = 1;
mplp.bed = regidx_init(optarg,NULL,NULL,0,NULL);
- if (!mplp.bed) { fprintf(bcftools_stderr, "bcftools mpileup: Could not read file \"%s\"", optarg); return 1; }
+ if (!mplp.bed) {
+ fprintf(bcftools_stderr, "bcftools mpileup: Could not read file \"%s\"", optarg);
+ goto err;
+ }
break;
case 'P': mplp.pl_list = strdup(optarg); break;
case 'p': mplp.flag |= MPLP_PER_SAMPLE; break;
else
mplp.indel_bias = 1/atof(optarg);
break;
+ case 27:
+ mplp.vs_ref = atof(optarg);
+ //if (mplp.vs_ref < 0) mplp.vs_ref = 0;
+ if (mplp.vs_ref > 1) mplp.vs_ref = 1;
+ break;
case 15: {
char *tmp;
mplp.indel_win_size = strtol(optarg,&tmp,10);
if ( *tmp ) error("Could not parse argument: --indel-size %s\n", optarg);
- if ( mplp.indel_win_size < 110 )
+ if ( mplp.indel_win_size < 20 )
{
- mplp.indel_win_size = 110;
+ mplp.indel_win_size = 20;
fprintf(bcftools_stderr,"Warning: running with --indel-size %d, the requested value is too small\n",mplp.indel_win_size);
}
}
break;
- case 20: mplp.indels_v20 = 1; break;
- case 21: mplp.write_index = 1; break;
+ case 20: mplp.indels_v20 = 1; mplp.edlib = 0; break;
+ case 'W':
+ if (!(mplp.write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
+ case 22: mplp.edlib = 1; mplp.indels_v20 = 0; break;
+ case 25: mplp.edlib = 0; break;
+ case 28:
+ mplp.seqQ_offset = atoi(optarg);
+ if (mplp.seqQ_offset < 100)
+ mplp.seqQ_offset = 100;
+ if (mplp.seqQ_offset > 200)
+ mplp.seqQ_offset = 200;
+ break;
+ case 23: mplp.del_bias = atof(optarg); break;
+ case 24: mplp.poly_mqual = 1; break;
+ case 26: mplp.poly_mqual = 0; break;
case 'A': use_orphan = 1; break;
case 'F': mplp.min_frac = atof(optarg); break;
case 'm': mplp.min_support = atoi(optarg); break;
case 'a':
if (optarg[0]=='?') {
list_annotations(bcftools_stderr);
- return 1;
+ goto err;
}
parse_format_flag(&mplp.fmt_flag,optarg);
break;
case 'M': mplp.max_read_len = atoi(optarg); break;
case 'X':
- if (strcasecmp(optarg, "pacbio-ccs") == 0) {
+ if (strcasecmp(optarg, "pacbio-ccs-1.18") == 0) {
mplp.min_frac = 0.1;
mplp.min_baseQ = 5;
mplp.max_baseQ = 50;
mplp.extQ = 1;
mplp.flag |= MPLP_REALN_PARTIAL;
mplp.max_read_len = 99999;
+
+ } else if (strcasecmp(optarg, "pacbio-ccs") == 0 ||
+ strcasecmp(optarg, "pacbio-ccs-1.20") == 0) {
+ mplp.min_frac = 0.1;
+ mplp.min_baseQ = 5;
+ mplp.max_baseQ = 50;
+ mplp.delta_baseQ = 10;
+ mplp.tandemQ = 300;
+ mplp.openQ = 25;
+ mplp.extQ = 1;
+ mplp.flag &= ~MPLP_REALN;
+ mplp.del_bias = 0.4;
+ mplp.indel_bias = 1/.9;
+ mplp.seqQ_offset = 118;
+ mplp.poly_mqual = 1;
+ mplp.edlib = 1;
+ mplp.vs_ref = 0.7;
+ mplp.indel_win_size = 80;
+
} else if (strcasecmp(optarg, "ont") == 0) {
- fprintf(bcftools_stderr, "For ONT it may be beneficial to also run bcftools call with "
+ fprintf(bcftools_stderr, "With old ONT data may be beneficial to also run bcftools call with "
"a higher -P, eg -P0.01 or -P 0.1\n");
mplp.min_baseQ = 5;
mplp.max_baseQ = 30;
mplp.flag &= ~MPLP_REALN;
mplp.flag |= MPLP_NO_INDEL;
+
+ } else if (strcasecmp(optarg, "ont-sup") == 0 ||
+ strcasecmp(optarg, "ont-sup-1.20") == 0) {
+ mplp.min_frac = 0.2;
+ mplp.min_baseQ = 1;
+ mplp.max_baseQ = 35;
+ mplp.delta_baseQ = 99;
+ mplp.openQ = 15;
+ mplp.extQ = 1;
+ mplp.flag &= ~MPLP_REALN;
+ mplp.max_read_len = 9999999;
+ mplp.del_bias = 0.4;
+ mplp.poly_mqual = 1;
+ mplp.edlib = 1;
+ // If we increase -h then we can increase bias denominator too
+ mplp.tandemQ = 110;
+ mplp.indel_bias = 1/0.7;
+ mplp.seqQ_offset = 130;
+ mplp.indel_win_size = 80;
+
+ } else if (strcasecmp(optarg, "ultima") == 0 ||
+ strcasecmp(optarg, "ultima-1.20") == 0) {
+ mplp.min_frac = 0.15;
+ mplp.min_baseQ = 1;
+ mplp.max_baseQ = 30;
+ mplp.delta_baseQ = 10;
+ mplp.openQ = 20;
+ mplp.extQ = 10;
+ mplp.tandemQ = 250;
+ mplp.flag &= ~MPLP_REALN;
+ mplp.del_bias = 0.3;
+ mplp.poly_mqual = 1;
+ mplp.edlib = 1;
+ mplp.indel_bias = 1/0.7;
+ mplp.seqQ_offset = 140;
+ mplp.vs_ref = 0.3;
+ mplp.indel_win_size = 80;
+
} else if (strcasecmp(optarg, "1.12") == 0) {
// 1.12 and earlier
mplp.min_frac = 0.002;
mplp.tandemQ = 100;
mplp.flag &= ~MPLP_REALN_PARTIAL;
mplp.flag |= MPLP_REALN;
- } else if (strcasecmp(optarg, "illumina") == 0) {
+
+ } else if (strcasecmp(optarg, "illumina-1.18") == 0) {
+ mplp.indel_win_size = 110;
+ mplp.flag |= MPLP_REALN_PARTIAL;
+
+ } else if (strcasecmp(optarg, "illumina") == 0 ||
+ strcasecmp(optarg, "illumina-1.20") == 0) {
+ mplp.edlib = 1;
+ mplp.indel_win_size = 110;
mplp.flag |= MPLP_REALN_PARTIAL;
+ mplp.indel_bias = 1;
+ mplp.seqQ_offset = 125;
+ //mplp.indel_win_size = 80; TEST?
+
+ } else if (strcasecmp(optarg, "bgi") == 0 ||
+ strcasecmp(optarg, "bgi-1.20") == 0) {
+ mplp.min_frac = 0.1;
+ mplp.edlib = 1;
+ mplp.indel_bias = 1;
+ mplp.seqQ_offset = 120;
+ mplp.flag |= MPLP_REALN_PARTIAL;
+ mplp.indel_win_size = 80;
+
+ } else if (strcasecmp(optarg, "list") == 0 ||
+ strcasecmp(optarg, "help") == 0) {
+ print_profiles();
+ goto err;
} else {
fprintf(bcftools_stderr, "Unknown configuration name '%s'\n"
- "Please choose from 1.12, illumina, pacbio-ccs or ont\n",
+ "Please use '-X list' to show available choices.\n",
optarg);
- return 1;
+ goto err;
}
break;
case 13: hts_srand48(atoi(optarg)); break;
break;
default:
fprintf(bcftools_stderr,"Invalid option: '%c'\n", c);
- return 1;
+ goto err;
}
}
if ( !(mplp.flag&MPLP_REALN) && mplp.flag&MPLP_REDO_BAQ )
{
fprintf(bcftools_stderr,"Error: The -B option cannot be combined with -E\n");
- return 1;
+ goto err;
}
if (use_orphan) mplp.flag &= ~MPLP_NO_ORPHAN;
if (argc == 1)
{
print_usage(bcftools_stderr, &mplp);
- return 1;
+ goto err;
}
if (!mplp.fai && !noref) {
fprintf(bcftools_stderr,"Error: mpileup requires the --fasta-ref option by default; use --no-reference to run without a fasta reference\n");
- return 1;
+ goto err;
}
- int ret,i;
+
if (file_list)
{
- if ( read_file_list(file_list,&nfiles,&fn) ) return 1;
+ if ( read_file_list(file_list,&nfiles,&fn) )
+ goto err;
mplp.files = fn;
mplp.nfiles = nfiles;
}
if (mplp.bed) regidx_destroy(mplp.bed);
if (mplp.bed_itr) regitr_destroy(mplp.bed_itr);
if (mplp.reg) regidx_destroy(mplp.reg);
+
+ err:
bam_smpl_destroy(mplp.bsmpl);
return ret;
}
y += len;
}
- else if ( op==BAM_CDEL )
+ else if ( op==BAM_CDEL || op==BAM_CREF_SKIP ) /* note: unsure about BAM_CREF_SKIP, don't have data to test */
{
if ( x>rcns->beg && x+len-1<=rcns->end )
{
}
y += len;
}
- else if ( op==BAM_CDEL )
+ else if ( op==BAM_CDEL || op==BAM_CREF_SKIP ) /* note: unsure about BAM_CREF_SKIP, don't have data to test */
{
if ( x>rcns->beg && x+len-1<=rcns->end )
{
*
* @param fname: input file name or NULL if regions will be added one-by-one via regidx_insert()
* @param parsef: regidx_parse_bed, regidx_parse_tab or see description of regidx_parse_f. If NULL,
- * the format will be autodected, currently either regidx_parse_tab (the default) or
+ * the format will be autodetected, currently either regidx_parse_tab (the default) or
* regidx_parse_bed (file must be named 'bed' or 'bed.gz') will be used. Note that
* the exact autodetection algorithm will change.
* @param freef: NULL or see description of regidx_parse_f
/* reheader.c -- reheader subcommand.
- Copyright (C) 2014-2022 Genome Research Ltd.
+ Copyright (C) 2014-2022,2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
typedef struct _args_t
{
char **argv, *fname, *samples_fname, *header_fname, *output_fname;
- char *fai_fname, *rm_tmpfile, *tmp_prefix;
+ char *fai_fname;
htsFile *fp;
+ faidx_t *fai;
htsFormat type;
htsThreadPool *threads;
int argc, n_threads;
kputs("/bcftools.XXXXXX", &prefix);
return prefix.s;
}
-static void update_from_fai(args_t *args)
+static void update_from_fai(faidx_t *fai, kstring_t *hdr_txt)
{
- if ( !strcmp("-",args->fname) )
- error("Cannot use the --fai option when reading from standard input.\n");
-
- faidx_t *fai = fai_load3(args->fai_fname,args->fai_fname,NULL,FAI_FASTA);
- if ( !fai ) error("Could not parse %s\n", args->fai_fname);
- args->rm_tmpfile = init_tmp_prefix(args->tmp_prefix);
- int fd = mkstemp(args->rm_tmpfile);
- if ( fd<0 ) error("Could not open a temporary file for writing: %s\n", args->rm_tmpfile);
-
- // get a template header: either from the original VCF or from --header
- char *ori_hdr_fname = args->header_fname ? args->header_fname : args->fname;
- htsFile *fp = hts_open(ori_hdr_fname,"r");
- if ( !fp ) error("Failed to open: %s\n", ori_hdr_fname);
- bcf_hdr_t *hdr = bcf_hdr_read(fp);
- if ( !hdr ) error("Failed to read the header: %s\n", ori_hdr_fname);
- hts_close(fp); // no need to check the return status here
-
- // put the header in a text buffer
- kstring_t hdr_txt_ori = {0,0,0}, hdr_txt_new = {0,0,0};
- bcf_hdr_format(hdr, 0, &hdr_txt_ori);
- bcf_hdr_destroy(hdr);
+ kstring_t hdr_txt_new = {0,0,0};
// update the existing contig lines and remove lines not present in the fai file
void *chr_seen = khash_str2int_init();
- char *tmp, *beg = hdr_txt_ori.s;
+ char *tmp, *beg = hdr_txt->s;
while ( beg && *beg )
{
tmp = strstr(beg, "\n##contig=<");
}
kputs(tmp+1,&hdr_txt_new);
- if ( write(fd, hdr_txt_new.s, hdr_txt_new.l)!=hdr_txt_new.l ) error("Failed to write %zu bytes to %s\n", hdr_txt_new.l,args->rm_tmpfile);
- if ( close(fd)!=0 ) error("Failed to close %s\n", args->rm_tmpfile);
- args->header_fname = args->rm_tmpfile;
+ // Switch the new header content for the old
+ free(hdr_txt->s);
+ memcpy(hdr_txt, &hdr_txt_new, sizeof(*hdr_txt));
- free(hdr_txt_ori.s);
- free(hdr_txt_new.s);
- fai_destroy(fai);
khash_str2int_destroy_free(chr_seen);
}
free(hdr.s); hdr.s = NULL; hdr.l = hdr.m = 0;
read_header_file(args->header_fname, &hdr);
}
+
+ if ( args->fai )
+ update_from_fai(args->fai, &hdr);
+
if ( samples )
{
set_samples(samples, nsamples, &hdr);
if ( bgzf_write(bgzf_out, hdr.s, hdr.l) < 0 ) error("Can't write BGZF header (code %d)\n", bgzf_out->errcode);
free(hdr.s);
- // Output all remainig data read with the header block
+ // Output all remaining data read with the header block
if ( fp->block_length - skip_until > 0 )
{
if ( bgzf_write(bgzf_out, buffer+skip_until, fp->block_length-skip_until)<0 ) error("Error: %d\n",fp->errcode);
free(hdr.s); hdr.s = NULL; hdr.l = hdr.m = 0;
read_header_file(args->header_fname, &hdr);
}
+
+ if ( args->fai )
+ update_from_fai(args->fai, &hdr);
+
if ( samples )
{
set_samples(samples, nsamples, &hdr);
free(htxt.s); htxt.s = NULL; htxt.l = htxt.m = 0;
read_header_file(args->header_fname, &htxt);
}
+
+ if ( args->fai )
+ update_from_fai(args->fai, &htxt);
+
if ( samples )
{
set_samples(samples, nsamples, &htxt);
fprintf(stderr, " -h, --header FILE new header\n");
fprintf(stderr, " -o, --output FILE write output to a file [standard output]\n");
fprintf(stderr, " -s, --samples FILE new sample names\n");
-#ifdef _WIN32
- fprintf(stderr, " -T, --temp-prefix PATH template for temporary file name [/bcftools.XXXXXX]\n");
-#else
- fprintf(stderr, " -T, --temp-prefix PATH template for temporary file name [/tmp/bcftools.XXXXXX]\n");
-#endif
+ fprintf(stderr, " -T, --temp-prefix PATH ignored; was template for temporary file name\n");
fprintf(stderr, " --threads INT use multithreading with <int> worker threads (BCF only) [0]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Example:\n");
switch (c)
{
case 1 : args->n_threads = strtol(optarg, 0, 0); break;
- case 'T': args->tmp_prefix = optarg; break;
+ case 'T': break; // unused - was temp file prefix
case 'f': args->fai_fname = optarg; break;
case 'o': args->output_fname = optarg; break;
case 's': args->samples_fname = optarg; break;
}
else args->fname = argv[optind];
- if ( args->fai_fname ) update_from_fai(args);
- if ( !args->samples_fname && !args->header_fname ) usage(args);
+ if ( args->fai_fname ) {
+ args->fai = fai_load3(args->fai_fname,args->fai_fname,NULL,FAI_FASTA);
+ if ( !args->fai ) error("Could not parse %s\n", args->fai_fname);
+ }
+ if ( !args->samples_fname && !args->header_fname && !args->fai) usage(args);
if ( !args->fname ) usage(args);
args->fp = hts_open(args->fname,"r");
else
reheader_bcf(args, args->type.compression==bgzf || args->type.compression==gzip);
- if ( args->rm_tmpfile )
- {
- unlink(args->rm_tmpfile);
- free(args->rm_tmpfile);
- }
+ if (args->fai)
+ fai_destroy(args->fai);
free(args);
return 0;
}
/* reheader.c -- reheader subcommand.
- Copyright (C) 2014-2022 Genome Research Ltd.
+ Copyright (C) 2014-2022,2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
typedef struct _args_t
{
char **argv, *fname, *samples_fname, *header_fname, *output_fname;
- char *fai_fname, *rm_tmpfile, *tmp_prefix;
+ char *fai_fname;
htsFile *fp;
+ faidx_t *fai;
htsFormat type;
htsThreadPool *threads;
int argc, n_threads;
kputs("/bcftools.XXXXXX", &prefix);
return prefix.s;
}
-static void update_from_fai(args_t *args)
+static void update_from_fai(faidx_t *fai, kstring_t *hdr_txt)
{
- if ( !strcmp("-",args->fname) )
- error("Cannot use the --fai option when reading from standard input.\n");
-
- faidx_t *fai = fai_load3(args->fai_fname,args->fai_fname,NULL,FAI_FASTA);
- if ( !fai ) error("Could not parse %s\n", args->fai_fname);
- args->rm_tmpfile = init_tmp_prefix(args->tmp_prefix);
- int fd = mkstemp(args->rm_tmpfile);
- if ( fd<0 ) error("Could not open a temporary file for writing: %s\n", args->rm_tmpfile);
-
- // get a template header: either from the original VCF or from --header
- char *ori_hdr_fname = args->header_fname ? args->header_fname : args->fname;
- htsFile *fp = hts_open(ori_hdr_fname,"r");
- if ( !fp ) error("Failed to open: %s\n", ori_hdr_fname);
- bcf_hdr_t *hdr = bcf_hdr_read(fp);
- if ( !hdr ) error("Failed to read the header: %s\n", ori_hdr_fname);
- hts_close(fp); // no need to check the return status here
-
- // put the header in a text buffer
- kstring_t hdr_txt_ori = {0,0,0}, hdr_txt_new = {0,0,0};
- bcf_hdr_format(hdr, 0, &hdr_txt_ori);
- bcf_hdr_destroy(hdr);
+ kstring_t hdr_txt_new = {0,0,0};
// update the existing contig lines and remove lines not present in the fai file
void *chr_seen = khash_str2int_init();
- char *tmp, *beg = hdr_txt_ori.s;
+ char *tmp, *beg = hdr_txt->s;
while ( beg && *beg )
{
tmp = strstr(beg, "\n##contig=<");
}
kputs(tmp+1,&hdr_txt_new);
- if ( write(fd, hdr_txt_new.s, hdr_txt_new.l)!=hdr_txt_new.l ) error("Failed to write %zu bytes to %s\n", hdr_txt_new.l,args->rm_tmpfile);
- if ( close(fd)!=0 ) error("Failed to close %s\n", args->rm_tmpfile);
- args->header_fname = args->rm_tmpfile;
+ // Switch the new header content for the old
+ free(hdr_txt->s);
+ memcpy(hdr_txt, &hdr_txt_new, sizeof(*hdr_txt));
- free(hdr_txt_ori.s);
- free(hdr_txt_new.s);
- fai_destroy(fai);
khash_str2int_destroy_free(chr_seen);
}
free(hdr.s); hdr.s = NULL; hdr.l = hdr.m = 0;
read_header_file(args->header_fname, &hdr);
}
+
+ if ( args->fai )
+ update_from_fai(args->fai, &hdr);
+
if ( samples )
{
set_samples(samples, nsamples, &hdr);
if ( bgzf_write(bgzf_out, hdr.s, hdr.l) < 0 ) error("Can't write BGZF header (code %d)\n", bgzf_out->errcode);
free(hdr.s);
- // Output all remainig data read with the header block
+ // Output all remaining data read with the header block
if ( fp->block_length - skip_until > 0 )
{
if ( bgzf_write(bgzf_out, buffer+skip_until, fp->block_length-skip_until)<0 ) error("Error: %d\n",fp->errcode);
free(hdr.s); hdr.s = NULL; hdr.l = hdr.m = 0;
read_header_file(args->header_fname, &hdr);
}
+
+ if ( args->fai )
+ update_from_fai(args->fai, &hdr);
+
if ( samples )
{
set_samples(samples, nsamples, &hdr);
free(htxt.s); htxt.s = NULL; htxt.l = htxt.m = 0;
read_header_file(args->header_fname, &htxt);
}
+
+ if ( args->fai )
+ update_from_fai(args->fai, &htxt);
+
if ( samples )
{
set_samples(samples, nsamples, &htxt);
fprintf(bcftools_stderr, " -h, --header FILE new header\n");
fprintf(bcftools_stderr, " -o, --output FILE write output to a file [standard output]\n");
fprintf(bcftools_stderr, " -s, --samples FILE new sample names\n");
-#ifdef _WIN32
- fprintf(bcftools_stderr, " -T, --temp-prefix PATH template for temporary file name [/bcftools.XXXXXX]\n");
-#else
- fprintf(bcftools_stderr, " -T, --temp-prefix PATH template for temporary file name [/tmp/bcftools.XXXXXX]\n");
-#endif
+ fprintf(bcftools_stderr, " -T, --temp-prefix PATH ignored; was template for temporary file name\n");
fprintf(bcftools_stderr, " --threads INT use multithreading with <int> worker threads (BCF only) [0]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Example:\n");
switch (c)
{
case 1 : args->n_threads = strtol(optarg, 0, 0); break;
- case 'T': args->tmp_prefix = optarg; break;
+ case 'T': break; // unused - was temp file prefix
case 'f': args->fai_fname = optarg; break;
case 'o': args->output_fname = optarg; break;
case 's': args->samples_fname = optarg; break;
}
else args->fname = argv[optind];
- if ( args->fai_fname ) update_from_fai(args);
- if ( !args->samples_fname && !args->header_fname ) usage(args);
+ if ( args->fai_fname ) {
+ args->fai = fai_load3(args->fai_fname,args->fai_fname,NULL,FAI_FASTA);
+ if ( !args->fai ) error("Could not parse %s\n", args->fai_fname);
+ }
+ if ( !args->samples_fname && !args->header_fname && !args->fai) usage(args);
if ( !args->fname ) usage(args);
args->fp = hts_open(args->fname,"r");
else
reheader_bcf(args, args->type.compression==bgzf || args->type.compression==gzip);
- if ( args->rm_tmpfile )
- {
- unlink(args->rm_tmpfile);
- free(args->rm_tmpfile);
- }
+ if (args->fai)
+ fai_destroy(args->fai);
free(args);
return 0;
}
/* str_finder.c -- Short Tandem Repeat finder.
Originally from Crumble (https://github.com/jkbonfield/crumble)
- Copyright (C) 2015-2016, 2021 Genome Research Ltd.
+ Copyright (C) 2015-2016, 2021-2022 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
return;
}
- // Find current and last occurence of repeated word.
+ // Find current and last occurrence of repeated word.
cp2 = &cons[pos+1];
// If unpadded, this is quicker: cp1 = &cons[pos+1-rlen];
* Returns a list of rep_ele structs holding the start,end tuples of repeats;
* NULL on failure.
*/
+rep_ele *find_STR64(char *cons, int len, int lower_only) {
+ int i, j;
+ uint64_t w = 0;
+ rep_ele *reps = NULL;
+
+ for (i = j = 0; i < len && j < 26; i++) {
+ if (cons[i] == '*') continue;
+
+ w <<= 2;
+ w |= cons[i];
+ //printf("%3d %c w=%08x\n", i, cons[i], w);
+ if (j>= 1 && (w&0x0003) == ((w>> 2)&0x0003))
+ add_rep(&reps, cons, len, i, 1, lower_only, w);
+ if (j>= 3 && (w&0x000f) == ((w>> 4)&0x000f))
+ add_rep(&reps, cons, len, i, 2, lower_only, w);
+ if (j>= 5 && (w&0x003f) == ((w>> 6)&0x003f))
+ add_rep(&reps, cons, len, i, 3, lower_only, w);
+ if (j>= 7 && (w&0x00ff) == ((w>> 8)&0x00ff))
+ add_rep(&reps, cons, len, i, 4, lower_only, w);
+ if (j>= 9 && (w&0x03ff) == ((w>>10)&0x03ff))
+ add_rep(&reps, cons, len, i, 5, lower_only, w);
+ if (j>=11 && (w&0x0fff) == ((w>>12)&0x0fff))
+ add_rep(&reps, cons, len, i, 6, lower_only, w);
+ if (j>=13 && (w&0x3fff) == ((w>>14)&0x3fff))
+ add_rep(&reps, cons, len, i, 7, lower_only, w);
+ if (j>=15 && (w&0xffff) == ((w>>16)&0xffff))
+ add_rep(&reps, cons, len, i, 8, lower_only, w);
+ if (j>=17 && (w&0x003ffff) == ((w>>18)&0x003ffff))
+ add_rep(&reps, cons, len, i, 9, lower_only, w);
+ if (j>=19 && (w&0x00fffff) == ((w>>20)&0x00fffff))
+ add_rep(&reps, cons, len, i,10, lower_only, w);
+ if (j>=21 && (w&0x03fffff) == ((w>>22)&0x03fffff))
+ add_rep(&reps, cons, len, i,11, lower_only, w);
+ if (j>=23 && (w&0x0ffffff) == ((w>>24)&0x0ffffff))
+ add_rep(&reps, cons, len, i,12, lower_only, w);
+ if (j>=24 && (w&0x3ffffff) == ((w>>26)&0x3ffffff))
+ add_rep(&reps, cons, len, i,13, lower_only, w);
+
+ j++;
+ }
+
+ for (; i < len; i++) {
+ if (cons[i] == '*') continue;
+
+ w <<= 2;
+ w |= cons[i];
+ //printf("%3d %c w=%08x\n", i, cons[i], w);
+ if ((w&0xfffffff) == ((w>>28)&0xfffffff))
+ add_rep(&reps, cons, len, i, 14, lower_only, w);
+ else if ((w&0x3ffffff) == ((w>>26)&0x3ffffff))
+ add_rep(&reps, cons, len, i, 13, lower_only, w);
+ else if ((w&0x0ffffff) == ((w>>24)&0x0ffffff))
+ add_rep(&reps, cons, len, i, 12, lower_only, w);
+ else if ((w&0x03fffff) == ((w>>22)&0x03fffff))
+ add_rep(&reps, cons, len, i, 11, lower_only, w);
+ else if ((w&0x00fffff) == ((w>>20)&0x00fffff))
+ add_rep(&reps, cons, len, i, 10, lower_only, w);
+ else if ((w&0x003ffff) == ((w>>18)&0x003ffff))
+ add_rep(&reps, cons, len, i, 9, lower_only, w);
+ else if ((w&0xffff) == ((w>>16)&0xffff))
+ add_rep(&reps, cons, len, i, 8, lower_only, w);
+ else if ((w&0x3fff) == ((w>>14)&0x3fff))
+ add_rep(&reps, cons, len, i, 7, lower_only, w);
+ else if ((w&0x0fff) == ((w>>12)&0x0fff))
+ add_rep(&reps, cons, len, i, 6, lower_only, w);
+ else if ((w&0x03ff) == ((w>>10)&0x03ff))
+ add_rep(&reps, cons, len, i, 5, lower_only, w);
+ else if ((w&0x00ff) == ((w>> 8)&0x00ff))
+ add_rep(&reps, cons, len, i, 4, lower_only, w);
+ else if ((w&0x003f) == ((w>> 6)&0x003f))
+ add_rep(&reps, cons, len, i, 3, lower_only, w);
+ else if ((w&0x000f) == ((w>> 4)&0x000f))
+ add_rep(&reps, cons, len, i, 2, lower_only, w);
+ else if ((w&0x0003) == ((w>> 2)&0x0003))
+ add_rep(&reps, cons, len, i, 1, lower_only, w);
+ }
+
+ return reps;
+}
+
rep_ele *find_STR(char *cons, int len, int lower_only) {
int i, j;
uint32_t w = 0;
w <<= 2;
w |= cons[i];
//printf("%3d %c w=%08x\n", i, cons[i], w);
- if ((w&0xffff) == ((w>>16)&0xffff))
+ if ((w&0xffff) == ((w>>16)&0xffff))
add_rep(&reps, cons, len, i, 8, lower_only, w);
- else if ((w&0x3fff) == ((w>>14)&0x3fff))
+ else if ((w&0x3fff) == ((w>>14)&0x3fff))
add_rep(&reps, cons, len, i, 7, lower_only, w);
- else if ((w&0x0fff) == ((w>>12)&0x0fff))
+ else if ((w&0x0fff) == ((w>>12)&0x0fff))
add_rep(&reps, cons, len, i, 6, lower_only, w);
- else if ((w&0x03ff) == ((w>>10)&0x03ff))
+ else if ((w&0x03ff) == ((w>>10)&0x03ff))
add_rep(&reps, cons, len, i, 5, lower_only, w);
- else if ((w&0x00ff) == ((w>> 8)&0x00ff))
+ else if ((w&0x00ff) == ((w>> 8)&0x00ff))
add_rep(&reps, cons, len, i, 4, lower_only, w);
- else if ((w&0x003f) == ((w>> 6)&0x003f))
+ else if ((w&0x003f) == ((w>> 6)&0x003f))
add_rep(&reps, cons, len, i, 3, lower_only, w);
- else if ((w&0x000f) == ((w>> 4)&0x000f))
+ else if ((w&0x000f) == ((w>> 4)&0x000f))
add_rep(&reps, cons, len, i, 2, lower_only, w);
- else if ((w&0x0003) == ((w>> 2)&0x0003))
+ else if ((w&0x0003) == ((w>> 2)&0x0003))
add_rep(&reps, cons, len, i, 1, lower_only, w);
}
/* str_finder.c -- Short Tandem Repeat finder.
Originally from Crumble (https://github.com/jkbonfield/crumble)
- Copyright (C) 2015-2016, 2021 Genome Research Ltd.
+ Copyright (C) 2015-2016, 2021-2022 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
return;
}
- // Find current and last occurence of repeated word.
+ // Find current and last occurrence of repeated word.
cp2 = &cons[pos+1];
// If unpadded, this is quicker: cp1 = &cons[pos+1-rlen];
* Returns a list of rep_ele structs holding the start,end tuples of repeats;
* NULL on failure.
*/
+rep_ele *find_STR64(char *cons, int len, int lower_only) {
+ int i, j;
+ uint64_t w = 0;
+ rep_ele *reps = NULL;
+
+ for (i = j = 0; i < len && j < 26; i++) {
+ if (cons[i] == '*') continue;
+
+ w <<= 2;
+ w |= cons[i];
+ //printf("%3d %c w=%08x\n", i, cons[i], w);
+ if (j>= 1 && (w&0x0003) == ((w>> 2)&0x0003))
+ add_rep(&reps, cons, len, i, 1, lower_only, w);
+ if (j>= 3 && (w&0x000f) == ((w>> 4)&0x000f))
+ add_rep(&reps, cons, len, i, 2, lower_only, w);
+ if (j>= 5 && (w&0x003f) == ((w>> 6)&0x003f))
+ add_rep(&reps, cons, len, i, 3, lower_only, w);
+ if (j>= 7 && (w&0x00ff) == ((w>> 8)&0x00ff))
+ add_rep(&reps, cons, len, i, 4, lower_only, w);
+ if (j>= 9 && (w&0x03ff) == ((w>>10)&0x03ff))
+ add_rep(&reps, cons, len, i, 5, lower_only, w);
+ if (j>=11 && (w&0x0fff) == ((w>>12)&0x0fff))
+ add_rep(&reps, cons, len, i, 6, lower_only, w);
+ if (j>=13 && (w&0x3fff) == ((w>>14)&0x3fff))
+ add_rep(&reps, cons, len, i, 7, lower_only, w);
+ if (j>=15 && (w&0xffff) == ((w>>16)&0xffff))
+ add_rep(&reps, cons, len, i, 8, lower_only, w);
+ if (j>=17 && (w&0x003ffff) == ((w>>18)&0x003ffff))
+ add_rep(&reps, cons, len, i, 9, lower_only, w);
+ if (j>=19 && (w&0x00fffff) == ((w>>20)&0x00fffff))
+ add_rep(&reps, cons, len, i,10, lower_only, w);
+ if (j>=21 && (w&0x03fffff) == ((w>>22)&0x03fffff))
+ add_rep(&reps, cons, len, i,11, lower_only, w);
+ if (j>=23 && (w&0x0ffffff) == ((w>>24)&0x0ffffff))
+ add_rep(&reps, cons, len, i,12, lower_only, w);
+ if (j>=24 && (w&0x3ffffff) == ((w>>26)&0x3ffffff))
+ add_rep(&reps, cons, len, i,13, lower_only, w);
+
+ j++;
+ }
+
+ for (; i < len; i++) {
+ if (cons[i] == '*') continue;
+
+ w <<= 2;
+ w |= cons[i];
+ //printf("%3d %c w=%08x\n", i, cons[i], w);
+ if ((w&0xfffffff) == ((w>>28)&0xfffffff))
+ add_rep(&reps, cons, len, i, 14, lower_only, w);
+ else if ((w&0x3ffffff) == ((w>>26)&0x3ffffff))
+ add_rep(&reps, cons, len, i, 13, lower_only, w);
+ else if ((w&0x0ffffff) == ((w>>24)&0x0ffffff))
+ add_rep(&reps, cons, len, i, 12, lower_only, w);
+ else if ((w&0x03fffff) == ((w>>22)&0x03fffff))
+ add_rep(&reps, cons, len, i, 11, lower_only, w);
+ else if ((w&0x00fffff) == ((w>>20)&0x00fffff))
+ add_rep(&reps, cons, len, i, 10, lower_only, w);
+ else if ((w&0x003ffff) == ((w>>18)&0x003ffff))
+ add_rep(&reps, cons, len, i, 9, lower_only, w);
+ else if ((w&0xffff) == ((w>>16)&0xffff))
+ add_rep(&reps, cons, len, i, 8, lower_only, w);
+ else if ((w&0x3fff) == ((w>>14)&0x3fff))
+ add_rep(&reps, cons, len, i, 7, lower_only, w);
+ else if ((w&0x0fff) == ((w>>12)&0x0fff))
+ add_rep(&reps, cons, len, i, 6, lower_only, w);
+ else if ((w&0x03ff) == ((w>>10)&0x03ff))
+ add_rep(&reps, cons, len, i, 5, lower_only, w);
+ else if ((w&0x00ff) == ((w>> 8)&0x00ff))
+ add_rep(&reps, cons, len, i, 4, lower_only, w);
+ else if ((w&0x003f) == ((w>> 6)&0x003f))
+ add_rep(&reps, cons, len, i, 3, lower_only, w);
+ else if ((w&0x000f) == ((w>> 4)&0x000f))
+ add_rep(&reps, cons, len, i, 2, lower_only, w);
+ else if ((w&0x0003) == ((w>> 2)&0x0003))
+ add_rep(&reps, cons, len, i, 1, lower_only, w);
+ }
+
+ return reps;
+}
+
rep_ele *find_STR(char *cons, int len, int lower_only) {
int i, j;
uint32_t w = 0;
w <<= 2;
w |= cons[i];
//printf("%3d %c w=%08x\n", i, cons[i], w);
- if ((w&0xffff) == ((w>>16)&0xffff))
+ if ((w&0xffff) == ((w>>16)&0xffff))
add_rep(&reps, cons, len, i, 8, lower_only, w);
- else if ((w&0x3fff) == ((w>>14)&0x3fff))
+ else if ((w&0x3fff) == ((w>>14)&0x3fff))
add_rep(&reps, cons, len, i, 7, lower_only, w);
- else if ((w&0x0fff) == ((w>>12)&0x0fff))
+ else if ((w&0x0fff) == ((w>>12)&0x0fff))
add_rep(&reps, cons, len, i, 6, lower_only, w);
- else if ((w&0x03ff) == ((w>>10)&0x03ff))
+ else if ((w&0x03ff) == ((w>>10)&0x03ff))
add_rep(&reps, cons, len, i, 5, lower_only, w);
- else if ((w&0x00ff) == ((w>> 8)&0x00ff))
+ else if ((w&0x00ff) == ((w>> 8)&0x00ff))
add_rep(&reps, cons, len, i, 4, lower_only, w);
- else if ((w&0x003f) == ((w>> 6)&0x003f))
+ else if ((w&0x003f) == ((w>> 6)&0x003f))
add_rep(&reps, cons, len, i, 3, lower_only, w);
- else if ((w&0x000f) == ((w>> 4)&0x000f))
+ else if ((w&0x000f) == ((w>> 4)&0x000f))
add_rep(&reps, cons, len, i, 2, lower_only, w);
- else if ((w&0x0003) == ((w>> 2)&0x0003))
+ else if ((w&0x0003) == ((w>> 2)&0x0003))
add_rep(&reps, cons, len, i, 1, lower_only, w);
}
/* str_finder.c -- Short Tandem Repeat finder.
Originally from Crumble (https://github.com/jkbonfield/crumble)
- Copyright (C) 2015-2016, 2021 Genome Research Ltd.
+ Copyright (C) 2015-2016, 2021, 2023 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
*/
rep_ele *find_STR(char *cons, int len, int lower_only);
+/* As above, but use a longer hash with longer STR elements found */
+rep_ele *find_STR64(char *cons, int len, int lower_only);
+
/*
* Returns an array of STR vs no-STR values.
* 0 => non repetitive.
* @file variantkey.h
* @brief VariantKey main functions.
*
- * The functions provided here allows to generate and process a 64 bit Unsigned Integer Keys for Human Genetic Variants.
+ * The functions provided here allow the generation and processing of a 64 bit Unsigned Integer Keys for Human Genetic Variants.
* The VariantKey is sortable for chromosome and position,
* and it is also fully reversible for variants with up to 11 bases between Reference and Alternate alleles.
* It can be used to sort, search and match variant-based data easily and very quickly.
{
if ((chrom[i] > '9') || (chrom[i] < '0'))
{
- return 0; // NA: a character that is not a numebr was found.
+ return 0; // NA: a character that is not a number was found.
}
v = ((v * 10) + (chrom[i] - '0'));
}
/* vcfannotate.c -- Annotate and edit VCF/BCF files.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
}
annot_col_t;
+typedef struct
+{
+ char *name; // column name
+ int ht_type; // type, one of BCF_HT_STR,BCF_HT_INT,BCF_HT_REAL
+ int icol; // index of the annotation column to use
+ union { // memory area with the current annotation value to pass to filter_test_ext
+ int i;
+ float f;
+ char *s;
+ };
+}
+ext_t;
+
// Logic of the filters: include or exclude sites which match the filters?
#define FLT_INCLUDE 1
#define FLT_EXCLUDE 2
regitr_t *tgt_itr;
int tgt_is_bed;
- filter_t *filter;
+ filter_t *filter, *filter_ext; // only one is initialized, the latter contains external values to set dynamically on the fly
char *filter_str;
int filter_logic; // include or exclude sites which match the filters? One of FLT_INCLUDE/FLT_EXCLUDE
int keep_sites;
convert_t *set_ids;
int set_ids_replace;
+ // external values for dynamic -i/-e expressions
+ int n_ext;
+ ext_t *ext;
+ void **ext_ptr;
+
int nsmpl_annot;
int *sample_map, nsample_map, sample_is_file; // map[idst] -> isrc
uint8_t *src_smpl_pld, *dst_smpl_pld; // for Number=G format fields
int argc, drop_header, record_cmd_line, tgts_is_vcf, mark_sites_logic, force, single_overlaps;
int columns_is_file, has_append_mode, pair_logic;
dbuf_t *header_lines;
+ bcf1_t *current_rec; // current record for local setters
}
args_t;
static int vcf_getter_id2str(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
{
char *str = *((char**)ptr);
- int len = strlen(rec->d.id);
+ int i, len = strlen(rec->d.id);
if ( len >= *mptr ) str = realloc(str, len+1);
- strcpy(str, rec->d.id);
+ for (i=0; i<len; i++)
+ str[i] = rec->d.id[i]==';' ? ',' : rec->d.id[i];
+ str[len] = 0;
*((char**)ptr) = str;
*mptr = len+1;
return len;
}
-static int vcf_getter_filter2str(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
+inline static int vcf_getter_filter2str_core(bcf_hdr_t *hdr, bcf1_t *rec, char **ptr, int *mptr)
{
+ if ( !(rec->unpacked & BCF_UN_FLT) ) bcf_unpack(rec, BCF_UN_FLT);
+
kstring_t str;
- str.s = *((char**)ptr);
+ str.s = *ptr;
str.m = *mptr;
str.l = 0;
{
for (i=0; i<rec->d.n_flt; i++)
{
- if (i) kputc(';', &str);
- kputs(bcf_hdr_int2id(args->tgts_hdr,BCF_DT_ID,rec->d.flt[i]), &str);
+ if (i) kputc(',', &str);
+ kputs(bcf_hdr_int2id(hdr,BCF_DT_ID,rec->d.flt[i]), &str);
}
}
else kputc('.', &str);
- *((char**)ptr) = str.s;
+ *ptr = str.s;
*mptr = str.m;
return str.l;
}
+static int vcf_getter_filter2str_local(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
+{
+ return vcf_getter_filter2str_core(args->hdr_out, args->current_rec, (char**)ptr, mptr);
+}
+static int vcf_getter_filter2str(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
+{
+ return vcf_getter_filter2str_core(args->tgts_hdr, rec, (char**)ptr, mptr);
+}
static int setter_filter(args_t *args, bcf1_t *line, annot_col_t *col, void *data)
{
if ( !data ) error("Error: the --merge-logic option cannot be used with FILTER (yet?)\n");
char *tmp;
int pos = strtol(tab->cols[col->icol], &tmp, 10);
if ( tmp==tab->cols[col->icol] )
- error("Could not parse ~POS at %s:%"PRId64" .. [%s]\n",bcf_seqname(args->hdr,line),(int64_t)line->pos+1,tab->cols[col->icol]);
+ error("Could not parse -POS at %s:%"PRId64" .. [%s]\n",bcf_seqname(args->hdr,line),(int64_t)line->pos+1,tab->cols[col->icol]);
line->pos = pos - 1;
return 0;
}
if (kh_exist(hash, k)) free((char*)kh_key(hash, k));
kh_clear(str2int, hash);
}
+static const char *escape_string(const char *str, char needle[], char **rmme, size_t *len)
+{
+ kstring_t tmp = {0,0,0};
+ const char *bp = str, *ep = str;
+ while ( *ep )
+ {
+ int i = 0;
+ while ( needle[i] && needle[i]!=*ep ) i++;
+ if ( !needle[i] ) { ep++; continue; }
+ kputsn(bp,ep-bp,&tmp);
+ ksprintf(&tmp,"%%%X",*ep);
+ bp = ++ep;
+ }
+ if ( !tmp.l )
+ {
+ *len = strlen(str);
+ return str;
+ }
+ kputs(bp,&tmp);
+ *len = tmp.l;
+ *rmme = tmp.s;
+ return tmp.s;
+}
static int setter_info_str(args_t *args, bcf1_t *line, annot_col_t *col, void *data)
{
if ( (col->replace & REPLACE_MISSING) && col->number!=BCF_VL_A && col->number!=BCF_VL_R )
if ( col->replace & SET_OR_APPEND ) col->merge_method=MM_UNIQUE;
annot_line_t *tab = (annot_line_t*) data;
+ const char *escaped = NULL;
+ char *rmme = NULL;
- int len = 0;
+ size_t len = 0;
if ( tab )
{
- len = strlen(tab->cols[col->icol]);
- if ( !len ) return 0;
- if ( len==1 && tab->cols[col->icol][0]=='.' && col->merge_method!=MM_APPEND_MISSING && !(col->replace & CARRY_OVER_MISSING) ) return 1;
+ char *str = tab->cols[col->icol];
+ if ( !str || !*str ) return 0;
+ if ( !str[1] && str[0]=='.' && col->merge_method!=MM_APPEND_MISSING && !(col->replace & CARRY_OVER_MISSING) ) return 1;
+ char needle[] = {';','=',0};
+ escaped = escape_string(tab->cols[col->icol],needle,&rmme,&len);
}
if ( col->merge_method!=MM_FIRST )
if ( col->merge_method==MM_UNIQUE )
{
if ( !col->mm_str_hash ) col->mm_str_hash = (khash_t(str2int)*)khash_str2int_init();
- if ( khash_str2int_has_key(col->mm_str_hash, tab->cols[col->icol]) ) return 1;
- khash_str2int_inc(col->mm_str_hash, strdup(tab->cols[col->icol]));
+ if ( khash_str2int_has_key(col->mm_str_hash, escaped) )
+ {
+ free(rmme);
+ return 1;
+ }
+ khash_str2int_inc(col->mm_str_hash, strdup(escaped));
}
if ( (col->replace & SET_OR_APPEND) && !col->mm_kstr.l )
}
if ( col->mm_kstr.l ) kputc(',',&col->mm_kstr);
- kputs(tab->cols[col->icol], &col->mm_kstr);
+ kputs(escaped, &col->mm_kstr);
+ free(rmme);
return 1;
}
-
if ( col->mm_kstr.l )
{
hts_expand(char,col->mm_kstr.l+1,args->mtmps,args->tmps);
memcpy(args->tmps,col->mm_kstr.s,col->mm_kstr.l+1);
}
else
+ {
+ free(rmme);
return 0;
+ }
// flush the line
if ( col->merge_method==MM_UNIQUE )
{
assert(tab);
hts_expand(char,len+1,args->mtmps,args->tmps);
- memcpy(args->tmps,tab->cols[col->icol],len+1);
-
+ memcpy(args->tmps,escaped,len+1);
if ( col->number==BCF_VL_A || col->number==BCF_VL_R )
return setter_ARinfo_string(args,line,col,tab->nals,tab->als);
}
-
- return bcf_update_info_string(args->hdr_out,line,col->hdr_key_dst,args->tmps);
+ int ret = bcf_update_info_string(args->hdr_out,line,col->hdr_key_dst,args->tmps);
+ free(rmme);
+ return ret;
}
static int vcf_setter_info_str(args_t *args, bcf1_t *line, annot_col_t *col, void *data)
{
if ( col->icol+args->nsmpl_annot > tab->ncols )
error("Incorrect number of values for %s at %s:%"PRId64"\n",col->hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ char needle[] = {':',0};
int ismpl;
for (ismpl=0; ismpl<args->nsmpl_annot; ismpl++)
- args->tmpp[ismpl] = tab->cols[col->icol + ismpl];
-
- return core_setter_format_str(args,line,col,args->tmpp);
+ {
+ size_t len;
+ char *rmme = NULL;
+ const char *str = escape_string(tab->cols[col->icol + ismpl],needle,&rmme,&len);
+ args->tmpp[ismpl] = rmme ? rmme : strdup(str);
+ }
+ int ret = core_setter_format_str(args,line,col,args->tmpp);
+ for (ismpl=0; ismpl<args->nsmpl_annot; ismpl++) free(args->tmpp[ismpl]);
+ return ret;
}
static int determine_ploidy(int nals, int *vals, int nvals1, uint8_t *smpl, int nsmpl)
{
kputsn(ss, se-ss, &str);
if ( !str.s[0] || !strcasecmp("-",str.s) ) ;
else if ( !strcasecmp("CHROM",str.s) ) args->chr_idx = icol;
- else if ( !strcasecmp("POS",str.s) ) args->beg_idx = icol;
+ else if ( !strcasecmp("POS",str.s) )
+ {
+ if ( replace==REPLACE_NON_MISSING && !args->tgts_is_vcf )
+ {
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = replace;
+ col->setter = setter_pos;
+ col->hdr_key_src = strdup(str.s);
+ col->hdr_key_dst = strdup(str.s);
+ args->match_end = icol;
+ }
+ else
+ args->beg_idx = icol;
+ }
else if ( !strcasecmp("FROM",str.s) || !strcasecmp("BEG",str.s) ) args->beg_idx = icol;
else if ( !strcasecmp("TO",str.s) || !strcasecmp("END",str.s) ) args->end_idx = icol;
else if ( !strcasecmp("REF",str.s) )
col->hdr_key_dst = strdup(str.s);
args->match_end = icol;
}
- else if ( !strcasecmp("~POS",str.s) && !args->tgts_is_vcf )
+ else if ( !strcasecmp("~POS",str.s) )
+ {
+ error("Error: the use of ~POS has been deprecated, use -POS to transfer the column POS.\n");
+ }
+ else if ( str.s[0]=='~' )
{
- if ( args->tgts_is_vcf ) error("Error: cannot use ~POS, position can be replaced only from a tab-delimited file\n");
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = MATCH_VALUE;
+ col->setter = NULL;
+ col->hdr_key_src = strdup(str.s+1);
+ }
+ else if ( !strcasecmp("-POS",str.s) && !args->tgts_is_vcf )
+ {
+ if ( args->tgts_is_vcf ) error("Error: cannot use -POS, position can be replaced only from a tab-delimited file\n");
args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
annot_col_t *col = &args->cols[args->ncols-1];
memset(col,0,sizeof(*col));
if ( bcf_hdr_id2type(args->tgts_hdr,BCF_HL_INFO,hdr_id)!=BCF_HT_STR )
error("Only Type=String tags can be used to annotate the ID column\n");
}
- else if ( (ptr=strstr(str.s,":=")) && !args->targets_fname )
+ else if ( (ptr=strstr(str.s,":=")) && (!args->targets_fname || !strncasecmp(ptr+2,"./",2)) )
{
*ptr = 0;
- rename_annots_push(args,ptr+2,str.s);
+ if ( !strncasecmp(str.s,"INFO/",5) && (!strcasecmp(ptr+2,"FILTER") || !strcasecmp(ptr+2,"./FILTER")) )
+ {
+ // -a not present and transferring filter, needs to be a local transfer
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = replace;
+ col->setter = vcf_setter_info_str;
+ col->getter = vcf_getter_filter2str_local;
+ col->hdr_key_src = strdup(ptr+2);
+ col->hdr_key_dst = strdup(str.s+5);
+ tmp.l = 0;
+ ksprintf(&tmp,"##INFO=<ID=%s,Number=1,Type=String,Description=\"Transferred FILTER column\">",col->hdr_key_dst);
+ bcf_hdr_append(args->hdr_out, tmp.s);
+ if (bcf_hdr_sync(args->hdr_out) < 0) error_errno("[%s] Failed to update header", __func__);
+ int hdr_id = bcf_hdr_id2int(args->hdr_out, BCF_DT_ID, col->hdr_key_dst);
+ col->number = bcf_hdr_id2length(args->hdr_out,BCF_HL_INFO,hdr_id);
+ }
+ else
+ rename_annots_push(args,ptr+2,str.s);
*ptr = ':';
}
else if ( !strcasecmp("FILTER",str.s) )
" (the annotation type is modified to \"Number=.\" and allele ordering is disregarded)\n");
fprintf(stderr,"Warning: the =INFO/TAG feature modifies the annotation to \"Number=.\" and disregards allele ordering\n");
}
+
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = replace;
+
int explicit_src_info = 0;
int explicit_dst_info = 0;
char *key_dst;
key_src[-2] = ':';
error("Did you mean \"FMT/%s\" rather than \"%s\"?\n",str.s,str.s);
}
+ else if ( !strcasecmp("FILTER",key_src) && args->tgts_is_vcf )
+ {
+ col->getter = vcf_getter_filter2str;
+ }
}
else
key_src = key_dst;
- args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
- annot_col_t *col = &args->cols[args->ncols-1];
- memset(col,0,sizeof(*col));
- col->icol = icol;
- col->replace = replace;
col->hdr_key_src = strdup(key_src);
col->hdr_key_dst = strdup(key_dst);
while ( *ptr && isspace(*ptr) ) ptr++;
if ( !*ptr ) { *rmme = ' '; error("Could not parse: %s\n", args->rename_annots_map[i]); }
if ( rename_annots_core(args, args->rename_annots_map[i], ptr) < 0 )
- error("Could not parse \"%s %s\", expected INFO, FORMAT, or FILTER prefix\n",args->rename_annots_map[i],ptr);
+ error("Cannot rename \"%s\" to \"%s\"\n",args->rename_annots_map[i],ptr);
}
}
static void rename_annots_push(args_t *args, char *src, char *dst)
ksprintf(&str,"%s %s",src,dst);
args->rename_annots_map[ args->rename_annots_nmap - 1 ] = str.s;
}
+static void init_filters(args_t *args)
+{
+ // Check if the -i/-e expressions contain external values that should be determined
+ // on the fly from the annotation file. The expressions can be given as
+ // TAG={NAME}
+ // TAG={str:NAME}
+ // TAG={int:NAME}
+ // TAG={float:NAME}
+ kstring_t str = {0,0,0};
+ char *src = strdup(args->filter_str);
+ int len = 0;
+ while (1)
+ {
+ char *beg = strchr(src+len,'{');
+ if ( !beg ) break;
+
+ // check if "{" appears inside quotes, in such case do not modify
+ char skip = 0;
+ char *tmp = src;
+ while ( tmp<beg )
+ {
+ if ( tmp[0]!='"' && tmp[0]!='\'' ) { tmp++; continue; }
+
+ // quote character found
+ int quote = tmp[0];
+ tmp++;
+ while ( *tmp && tmp[0]!=quote ) tmp++;
+ if ( !*tmp ) error("Could not parse the expression: %s\n",args->filter_str); // unbalanced quotation; todo: check for escape char
+ len = tmp - src + 1;
+ skip = 1;
+ }
+ if ( skip ) continue;
+
+ char *end = ++beg;
+ while ( *end && *end!='}' ) end++;
+ if ( !*end ) error("Could not parse the expression: %s\n",args->filter_str);
+ *end = 0;
+
+ // explicit typing?
+ int type = -1;
+ tmp = beg;
+ while ( *tmp && *tmp!=':' ) tmp++;
+ if ( *tmp )
+ {
+ *tmp = 0;
+ if ( !strcasecmp(beg,"str") ) type = BCF_HT_STR;
+ else if ( !strcasecmp(beg,"int") ) type = BCF_HT_INT;
+ else if ( !strcasecmp(beg,"float") ) type = BCF_HT_REAL;
+ }
+ args->n_ext++;
+ args->ext = (ext_t*)realloc(args->ext,sizeof(*args->ext)*args->n_ext);
+ ext_t *ext = &args->ext[args->n_ext-1];
+ ext->ht_type = type;
+ ext->name = strdup(beg);
+ if ( beg-1 > src ) kputsn(src,beg-1-src,&str);
+ if ( type==-1 ) kputs("{}",&str);
+ else if ( type==BCF_HT_STR ) kputs("{str}",&str);
+ else if ( type==BCF_HT_INT ) kputs("{int}",&str);
+ else if ( type==BCF_HT_REAL ) kputs("{float}",&str);
+ len = str.l;
+ kputs(end+1,&str);
+ free(src);
+ src = strdup(str.s);
+ str.l = 0;
+ }
+ args->filter = filter_init(args->hdr, src);
+ free(src);
+ free(str.s);
+
+ int i,j,n_ext;
+ const int *ext_type = filter_ext_types(args->filter, &n_ext);
+ if ( n_ext != args->n_ext )
+ error("Failed to parse the expression, unexpected number of dynamic variables (%d vs %d): %s\n",n_ext,args->n_ext,args->filter_str);
+
+ if ( !args->n_ext ) return;
+
+ if ( !args->tgts )
+ error("Error: dynamic variables in -i/-e expressions can be currently used only with tab-delimited file, not with VCF (todo)\n");
+
+ // contains external values
+ args->ext_ptr = malloc(sizeof(*args->ext_ptr)*args->n_ext);
+ for (i=0; i<args->n_ext; i++) args->ext[i].ht_type = ext_type[i];
+ args->filter_ext = args->filter;
+ args->filter = NULL;
+
+ // set the column idx
+ if ( args->ncols )
+ {
+ for (i=0; i<args->n_ext; i++)
+ {
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( strcmp(args->ext[i].name,args->cols[j].hdr_key_src) ) continue;
+ args->ext[i].icol = args->cols[j].icol;
+ break;
+ }
+ if ( j==args->ncols ) error("No such column: %s\n",args->ext[i].name);
+ }
+ }
+}
static void init_data(args_t *args)
{
args->vcmp = vcmp_init();
if ( args->filter_str )
- args->filter = filter_init(args->hdr, args->filter_str);
+ init_filters(args);
if ( args->mark_sites )
{
if ( args->n_threads )
hts_set_opt(args->out_fh, HTS_OPT_THREAD_POOL, args->files->p);
if ( bcf_hdr_write(args->out_fh, args->hdr_out)!=0 ) error("[%s] Error: failed to write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr,args->output_fname,
+ &args->index_fn, args->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
}
}
static void destroy_data(args_t *args)
{
int i;
+ for (i=0; i<args->n_ext; i++)
+ {
+ free(args->ext[i].name);
+ if ( args->ext[i].ht_type!=BCF_HT_STR ) continue;
+ }
+ free(args->ext_ptr);
+ free(args->ext);
for (i=0; i<args->nrm; i++) free(args->rm[i].key);
free(args->rm);
if ( args->hdr_out ) bcf_hdr_destroy(args->hdr_out);
free(args->dst_smpl_pld);
if ( args->set_ids )
convert_destroy(args->set_ids);
- if ( args->filter )
- filter_destroy(args->filter);
+ if ( args->filter ) filter_destroy(args->filter);
+ if ( args->filter_ext ) filter_destroy(args->filter_ext);
if (args->out_fh)
{
if ( args->write_index )
}
else i++;
}
- if ( args->ref_idx==-1 && args->nalines ) return;
+ if ( !args->filter_ext && args->ref_idx==-1 && args->nalines ) return;
while ( !bcf_sr_regions_overlap(args->tgts, bcf_seqname(args->hdr,line), start_pos,end_pos) )
{
tmp->start = args->tgts->start;
tmp->end = args->tgts->end;
parse_annot_line(args, args->tgts->line.s, tmp);
- if ( args->ref_idx != -1 )
+ if ( args->filter_ext || args->ref_idx != -1 )
{
int iseq = args->tgts->iseq;
if ( bcf_sr_regions_next(args->tgts)<0 || args->tgts->iseq!=iseq ) break;
}
return 0;
}
-static void annotate(args_t *args, bcf1_t *line)
+static int annotate_from_regidx(args_t *args, bcf1_t *line)
{
- int i, j;
- for (i=0; i<args->nrm; i++)
- args->rm[i].handler(args, line, &args->rm[i]);
-
+ int j;
int has_overlap = 0;
- if ( args->tgt_idx )
+
+ for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
+ if ( regidx_overlap(args->tgt_idx, bcf_seqname(args->hdr,line),line->pos,line->pos+line->rlen-1, args->tgt_itr) )
{
- for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
- if ( regidx_overlap(args->tgt_idx, bcf_seqname(args->hdr,line),line->pos,line->pos+line->rlen-1, args->tgt_itr) )
+ hts_pos_t vcf_end = line->pos + line->rlen - 1;
+ while ( regitr_overlap(args->tgt_itr) )
{
- hts_pos_t vcf_end = line->pos + line->rlen - 1;
- while ( regitr_overlap(args->tgt_itr) )
- {
- annot_line_t *tmp = &args->alines[0];
- tmp->rid = line->rid;
- tmp->start = args->tgt_itr->beg;
- tmp->end = args->tgt_itr->end;
+ annot_line_t *tmp = &args->alines[0];
+ tmp->rid = line->rid;
+ tmp->start = args->tgt_itr->beg;
+ tmp->end = args->tgt_itr->end;
- // Check min overlap
- int len_ann = tmp->end - tmp->start + 1;
- int len_vcf = line->rlen;
- int isec = (tmp->end < vcf_end ? tmp->end : vcf_end) - (tmp->start > line->pos ? tmp->start : line->pos) + 1;
- assert( isec > 0 );
- if ( args->min_overlap_ann && args->min_overlap_ann > (float)isec/len_ann ) continue;
- if ( args->min_overlap_vcf && args->min_overlap_vcf > (float)isec/len_vcf ) continue;
+ // Check min overlap
+ int len_ann = tmp->end - tmp->start + 1;
+ int len_vcf = line->rlen;
+ int isec = (tmp->end < vcf_end ? tmp->end : vcf_end) - (tmp->start > line->pos ? tmp->start : line->pos) + 1;
+ assert( isec > 0 );
+ if ( args->min_overlap_ann && args->min_overlap_ann > (float)isec/len_ann ) continue;
+ if ( args->min_overlap_vcf && args->min_overlap_vcf > (float)isec/len_vcf ) continue;
- parse_annot_line(args, regitr_payload(args->tgt_itr,char*), tmp);
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].done==1 ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],tmp);
- if ( ret < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- if ( ret==0 )
- args->cols[j].done = 1;
- has_overlap = 1;
- }
+ parse_annot_line(args, regitr_payload(args->tgt_itr,char*), tmp);
+
+ // If a plain BED file is provided and we are asked to just mark overlapping sites, there are
+ // no additional columns. Not sure if there can be any side effects for ill-formatted BED files
+ // with variable number of columns
+ if ( !args->ncols && args->mark_sites ) has_overlap = 1;
+
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],tmp);
+ if ( ret < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ if ( ret==0 )
+ args->cols[j].done = 1;
+ has_overlap = 1;
}
}
- for (j=0; j<args->ncols; j++)
+ }
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
+ if ( !args->cols[j].setter ) continue;
+ if ( args->cols[j].setter(args,line,&args->cols[j],NULL) < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ }
+ return has_overlap;
+}
+static int pass_filter_test_ext(args_t *args, bcf1_t *line, annot_line_t *ann)
+{
+ char *tmp;
+ int i;
+ for (i=0; i<args->n_ext; i++)
+ {
+ int j = args->ext[i].icol;
+ if ( args->ext[i].ht_type==BCF_HT_STR ) args->ext_ptr[i] = args->ext[i].s = ann->cols[j];
+ else if ( args->ext[i].ht_type==BCF_HT_INT )
{
- if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
- if ( !args->cols[j].setter ) continue;
- if ( args->cols[j].setter(args,line,&args->cols[j],NULL) < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ args->ext[i].i = strtol(ann->cols[j],&tmp,10);
+ if ( *tmp )
+ {
+ if ( strcmp(".",ann->cols[j]) ) error("Error: could not parse the annotation file, expected an integer, found \"%s\"\n",ann->cols[j]);
+ args->ext_ptr[i] = NULL;
+ }
+ else
+ args->ext_ptr[i] = &args->ext[i].i;
+ }
+ else if ( args->ext[i].ht_type==BCF_HT_REAL )
+ {
+ args->ext[i].f = strtod(ann->cols[j],&tmp);
+ if ( *tmp )
+ {
+ if ( strcmp(".",ann->cols[j]) ) error("Error: could not parse the annotation file, expected a float, found \"%s\"\n",ann->cols[j]);
+ args->ext_ptr[i] = NULL;
+ }
+ else
+ args->ext_ptr[i] = &args->ext[i].f;
}
}
- else if ( args->tgts )
- {
- // Buffer annotation lines. When multiple ALT alleles are present in the annotation file, at least one
- // must match some of the VCF alleles. If the append-missing mode is set (and REF+ALT is requested), the
- // buffered lines will annotate the VCF respecting the order in ALT and when no matching line is found
- // for an ALT, missing value is appended instead.
- int end_pos = line->pos + line->rlen - 1;
- buffer_annot_lines(args, line, line->pos, end_pos);
+ int pass = filter_test_ext(args->filter_ext,line,NULL,(const void**)args->ext_ptr);
+ if ( args->filter_logic==FLT_EXCLUDE ) pass = pass ? 0 : 1;
+ return pass;
+}
+static int annotate_from_tab(args_t *args, bcf1_t *line)
+{
+ int i,j;
+ int has_overlap = 0;
+
+ // Buffer annotation lines. When multiple ALT alleles are present in the annotation file, at least one
+ // must match some of the VCF alleles. If the append-missing mode is set (and REF+ALT is requested), the
+ // buffered lines will annotate the VCF respecting the order in ALT and when no matching line is found
+ // for an ALT, missing value is appended instead.
+ int end_pos = line->pos + line->rlen - 1;
+ buffer_annot_lines(args, line, line->pos, end_pos);
- args->nsrt_alines = 0;
- hts_expand(uint32_t,args->nalines,args->msrt_alines,args->srt_alines);
- if ( args->nalines >= 0xffff || line->n_allele >= 0xffff )
- error("Error: too many alleles or annotation lines in the buffer at %s:%"PRId64" (todo:skip?)\n",bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ args->nsrt_alines = 0;
+ hts_expand(uint32_t,args->nalines,args->msrt_alines,args->srt_alines);
+ if ( args->nalines >= 0xffff || line->n_allele >= 0xffff )
+ error("Error: too many alleles or annotation lines in the buffer at %s:%"PRId64" (todo:skip?)\n",bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- kstring_t match_end = {0,0,0};
- if ( args->match_end>=0 && bcf_get_info_int32(args->hdr,line,"END",&args->tmpi,&args->mtmpi)==1 )
- kputw(args->tmpi[0],&match_end);
+ kstring_t match_end = {0,0,0};
+ if ( args->match_end>=0 && bcf_get_info_int32(args->hdr,line,"END",&args->tmpi,&args->mtmpi)==1 )
+ kputw(args->tmpi[0],&match_end);
- // Find matching lines
- for (i=0; i<args->nalines; i++)
+ // Find matching lines
+ for (i=0; i<args->nalines; i++)
+ {
+ if ( line->pos > args->alines[i].end || end_pos < args->alines[i].start ) continue;
+ if ( args->ref_idx != -1 ) // REF+ALT matching requested
{
- if ( line->pos > args->alines[i].end || end_pos < args->alines[i].start ) continue;
- if ( args->ref_idx != -1 ) // REF+ALT matching requested
+ if ( line->pos!=args->alines[i].start || vcmp_set_ref(args->vcmp, line->d.allele[0], args->alines[i].als[0]) < 0 ) continue; // refs are not compatible
+ for (j=1; j<args->alines[i].nals; j++)
{
- if ( line->pos!=args->alines[i].start || vcmp_set_ref(args->vcmp, line->d.allele[0], args->alines[i].als[0]) < 0 ) continue; // refs are not compatible
- for (j=1; j<args->alines[i].nals; j++)
+ int ialt;
+ if ( line->n_allele==1 && args->alines[i].als[j][0]=='.' && args->alines[i].als[j][1]==0 ) // match: no ALT allele in VCF and annot file has "."
+ ialt = 0;
+ else
{
- int ialt;
- if ( line->n_allele==1 && args->alines[i].als[j][0]=='.' && args->alines[i].als[j][1]==0 ) // match: no ALT allele in VCF and annot file has "."
- ialt = 0;
- else
- {
- ialt = vcmp_find_allele(args->vcmp, line->d.allele+1, line->n_allele - 1, args->alines[i].als[j]);
- if ( ialt < 0 ) continue;
- ialt++;
- }
- if ( args->match_id>=0 && !strstr_match(line->d.id,args->alines[i].cols[args->match_id]) ) continue;
- if ( args->match_end>=0 && match_end.l && strcmp(match_end.s,args->alines[i].cols[args->match_end]) ) continue;
- args->srt_alines[args->nsrt_alines++] = (ialt<<16) | i;
- has_overlap = 1;
- break;
+ ialt = vcmp_find_allele(args->vcmp, line->d.allele+1, line->n_allele - 1, args->alines[i].als[j]);
+ if ( ialt < 0 ) continue;
+ ialt++;
}
+ if ( args->match_id>=0 && !strstr_match(line->d.id,args->alines[i].cols[args->match_id]) ) continue;
+ if ( args->match_end>=0 && match_end.l && strcmp(match_end.s,args->alines[i].cols[args->match_end]) ) continue;
+ if ( args->filter_ext && !pass_filter_test_ext(args,line,&args->alines[i]) ) continue;
+ args->srt_alines[args->nsrt_alines++] = (ialt<<16) | i;
+ has_overlap = 1;
+ break;
}
- else // overlap, REF+ALT matching not requested
+ }
+ else if ( args->filter_ext )
+ {
+ if ( pass_filter_test_ext(args,line,&args->alines[i]) )
{
args->srt_alines[args->nsrt_alines++] = (0xffff<<16) | i;
has_overlap = 1;
}
}
+ else // overlap, REF+ALT matching not requested
+ {
+ args->srt_alines[args->nsrt_alines++] = (0xffff<<16) | i;
+ has_overlap = 1;
+ }
+ }
- free(match_end.s);
+ free(match_end.s);
+ if ( !has_overlap && args->filter_ext && !args->keep_sites ) return has_overlap;
- // Sort lines if needed
+ // Sort lines if needed
+ if ( args->has_append_mode )
+ {
+ // insertion sort by VCF ALT index (top bits) and alines index (low bits)
+ uint32_t tmp;
+ for (i=1; i<args->nsrt_alines; i++)
+ for (j=i; j>0 && args->srt_alines[j] < args->srt_alines[j-1]; j--)
+ tmp = args->srt_alines[j], args->srt_alines[j] = args->srt_alines[j-1], args->srt_alines[j-1] = tmp;
+ }
+ // Annotate
+ for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
+ int ialt_exp = 1;
+ for (i=0; i<args->nsrt_alines; i++)
+ {
+ int ialt = args->srt_alines[i] >> 16;
+ int ilin = args->srt_alines[i] & 0xffff;
if ( args->has_append_mode )
{
- // insertion sort by VCF ALT index (top bits) and alines index (low bits)
- uint32_t tmp;
- for (i=1; i<args->nsrt_alines; i++)
- for (j=i; j>0 && args->srt_alines[j] < args->srt_alines[j-1]; j--)
- tmp = args->srt_alines[j], args->srt_alines[j] = args->srt_alines[j-1], args->srt_alines[j-1] = tmp;
- }
- // Annotate
- for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
- int ialt_exp = 1;
- for (i=0; i<args->nsrt_alines; i++)
- {
- int ialt = args->srt_alines[i] >> 16;
- int ilin = args->srt_alines[i] & 0xffff;
- if ( args->has_append_mode )
+ if ( ialt_exp > ialt ) continue; // multiple annotation lines for the same position
+ if ( ialt_exp < ialt )
{
- if ( ialt_exp > ialt ) continue; // multiple annotation lines for the same position
- if ( ialt_exp < ialt )
- {
- // REF+ALT matching requested, append-missing mode: insert "." if no annotation line was found for the ALT
- while ( ialt_exp++ < ialt )
- {
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].merge_method != MM_APPEND_MISSING ) continue;
- if ( args->cols[j].done==1 ) continue;
- if ( !args->cols[j].setter ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],args->aline_missing);
- if ( ret < 0 )
- error("fixme: Could not set missing %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- if ( ret==0 )
- args->cols[j].done = 1;
- }
- }
- }
- }
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].done==1 ) continue;
- if ( !args->cols[j].setter ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],&args->alines[ilin]);
- if ( ret < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- if ( ret==0 )
- args->cols[j].done = 1;
- }
- ialt_exp = ialt + 1;
- }
- if ( args->nsrt_alines )
- {
- // In the append-missing mode fill missing values to all trailing ALTs, but only if at least one
- // record was found. Otherwise leave the row will be left without annotation.
- if ( args->has_append_mode && ialt_exp < line->n_allele )
- {
- while ( ialt_exp++ < line->n_allele )
+ // REF+ALT matching requested, append-missing mode: insert "." if no annotation line was found for the ALT
+ while ( ialt_exp++ < ialt )
{
for (j=0; j<args->ncols; j++)
{
}
}
}
- // Flush
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
- if ( !args->cols[j].setter ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],NULL);
- if ( ret < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- }
}
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 ) continue;
+ if ( !args->cols[j].setter ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],&args->alines[ilin]);
+ if ( ret < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ if ( ret==0 )
+ args->cols[j].done = 1;
+ }
+ ialt_exp = ialt + 1;
}
- else if ( args->files->nreaders == 2 )
+ if ( args->nsrt_alines )
{
- if ( bcf_sr_has_line(args->files,1) )
+ // In the append-missing mode fill missing values to all trailing ALTs, but only if at least one
+ // record was found. Otherwise leave the row will be left without annotation.
+ if ( args->has_append_mode && ialt_exp < line->n_allele )
{
- bcf1_t *aline = bcf_sr_get_line(args->files,1);
- for (j=0; j<args->ncols; j++)
+ while ( ialt_exp++ < line->n_allele )
{
- if ( !args->cols[j].setter ) continue;
- if ( args->cols[j].setter(args,line,&args->cols[j],aline) )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].merge_method != MM_APPEND_MISSING ) continue;
+ if ( args->cols[j].done==1 ) continue;
+ if ( !args->cols[j].setter ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],args->aline_missing);
+ if ( ret < 0 )
+ error("fixme: Could not set missing %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ if ( ret==0 )
+ args->cols[j].done = 1;
+ }
}
-
- has_overlap = 1;
}
+ // Flush
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
+ if ( !args->cols[j].setter ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],NULL);
+ if ( ret < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ }
+ }
+ return has_overlap;
+}
+static int annotate_from_vcf(args_t *args, bcf1_t *line)
+{
+ if ( !bcf_sr_has_line(args->files,1) ) return 0;
+ int j;
+ bcf1_t *aline = bcf_sr_get_line(args->files,1);
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( !args->cols[j].setter ) continue;
+ if ( args->cols[j].setter(args,line,&args->cols[j],aline) )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
}
+ return 1;
+}
+static int annotate_from_self(args_t *args, bcf1_t *line)
+{
+ int j;
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( !args->cols[j].setter ) continue;
+ if ( args->cols[j].setter(args,line,&args->cols[j],NULL) )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ }
+ return 0;
+}
+static int annotate_line(args_t *args, bcf1_t *line)
+{
+ args->current_rec = line;
+
+ int i;
+ for (i=0; i<args->nrm; i++)
+ args->rm[i].handler(args, line, &args->rm[i]);
+
+ int has_overlap = 0;
+ if ( args->tgt_idx )
+ has_overlap = annotate_from_regidx(args,line);
+
+ else if ( args->tgts )
+ has_overlap = annotate_from_tab(args,line);
+
+ else if ( args->files->nreaders == 2 )
+ has_overlap = annotate_from_vcf(args,line);
+
+ else if ( args->ncols )
+ has_overlap = annotate_from_self(args,line);
+
if ( args->set_ids )
{
args->tmpks.l = 0;
else
bcf_update_info_flag(args->hdr_out,line,args->mark_sites,NULL,has_overlap?0:1);
}
+
+ return has_overlap;
}
static void usage(args_t *args)
fprintf(stderr, " --single-overlaps Keep memory low by avoiding complexities arising from handling multiple overlapping intervals\n");
fprintf(stderr, " -x, --remove LIST List of annotations (e.g. ID,INFO/DP,FORMAT/DP,FILTER) to remove (or keep with \"^\" prefix). See man page for details\n");
fprintf(stderr, " --threads INT Number of extra output compression threads [0]\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Examples:\n");
fprintf(stderr, " http://samtools.github.io/bcftools/howtos/annotate.html\n");
{"min-overlap",required_argument,NULL,12},
{"no-version",no_argument,NULL,8},
{"force",no_argument,NULL,'f'},
- {"write-index",no_argument,NULL,13},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "h:H:?o:O:r:R:a:x:c:C:i:e:S:s:I:m:kl:f",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h:H:?o:O:r:R:a:x:c:C:i:e:S:s:I:m:kl:fW::",loptions,NULL)) >= 0)
{
switch (c) {
case 'f': args->force = 1; break;
case 10 : args->single_overlaps = 1; break;
case 11 : args->rename_annots = optarg; break;
case 12 : args->min_overlap_str = optarg; break;
- case 13 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
}
continue;
}
}
- annotate(args, line);
+ int keep = annotate_line(args, line);
+ if ( args->filter_ext && !args->keep_sites && !keep ) continue;
if ( bcf_write1(args->out_fh, args->hdr_out, line)!=0 ) error("[%s] Error: failed to write to %s\n", __func__,args->output_fname);
}
destroy_data(args);
/* vcfannotate.c -- Annotate and edit VCF/BCF files.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
}
annot_col_t;
+typedef struct
+{
+ char *name; // column name
+ int ht_type; // type, one of BCF_HT_STR,BCF_HT_INT,BCF_HT_REAL
+ int icol; // index of the annotation column to use
+ union { // memory area with the current annotation value to pass to filter_test_ext
+ int i;
+ float f;
+ char *s;
+ };
+}
+ext_t;
+
// Logic of the filters: include or exclude sites which match the filters?
#define FLT_INCLUDE 1
#define FLT_EXCLUDE 2
regitr_t *tgt_itr;
int tgt_is_bed;
- filter_t *filter;
+ filter_t *filter, *filter_ext; // only one is initialized, the latter contains external values to set dynamically on the fly
char *filter_str;
int filter_logic; // include or exclude sites which match the filters? One of FLT_INCLUDE/FLT_EXCLUDE
int keep_sites;
convert_t *set_ids;
int set_ids_replace;
+ // external values for dynamic -i/-e expressions
+ int n_ext;
+ ext_t *ext;
+ void **ext_ptr;
+
int nsmpl_annot;
int *sample_map, nsample_map, sample_is_file; // map[idst] -> isrc
uint8_t *src_smpl_pld, *dst_smpl_pld; // for Number=G format fields
int argc, drop_header, record_cmd_line, tgts_is_vcf, mark_sites_logic, force, single_overlaps;
int columns_is_file, has_append_mode, pair_logic;
dbuf_t *header_lines;
+ bcf1_t *current_rec; // current record for local setters
}
args_t;
static int vcf_getter_id2str(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
{
char *str = *((char**)ptr);
- int len = strlen(rec->d.id);
+ int i, len = strlen(rec->d.id);
if ( len >= *mptr ) str = realloc(str, len+1);
- strcpy(str, rec->d.id);
+ for (i=0; i<len; i++)
+ str[i] = rec->d.id[i]==';' ? ',' : rec->d.id[i];
+ str[len] = 0;
*((char**)ptr) = str;
*mptr = len+1;
return len;
}
-static int vcf_getter_filter2str(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
+inline static int vcf_getter_filter2str_core(bcf_hdr_t *hdr, bcf1_t *rec, char **ptr, int *mptr)
{
+ if ( !(rec->unpacked & BCF_UN_FLT) ) bcf_unpack(rec, BCF_UN_FLT);
+
kstring_t str;
- str.s = *((char**)ptr);
+ str.s = *ptr;
str.m = *mptr;
str.l = 0;
{
for (i=0; i<rec->d.n_flt; i++)
{
- if (i) kputc(';', &str);
- kputs(bcf_hdr_int2id(args->tgts_hdr,BCF_DT_ID,rec->d.flt[i]), &str);
+ if (i) kputc(',', &str);
+ kputs(bcf_hdr_int2id(hdr,BCF_DT_ID,rec->d.flt[i]), &str);
}
}
else kputc('.', &str);
- *((char**)ptr) = str.s;
+ *ptr = str.s;
*mptr = str.m;
return str.l;
}
+static int vcf_getter_filter2str_local(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
+{
+ return vcf_getter_filter2str_core(args->hdr_out, args->current_rec, (char**)ptr, mptr);
+}
+static int vcf_getter_filter2str(args_t *args, bcf1_t *rec, annot_col_t *col, void **ptr, int *mptr)
+{
+ return vcf_getter_filter2str_core(args->tgts_hdr, rec, (char**)ptr, mptr);
+}
static int setter_filter(args_t *args, bcf1_t *line, annot_col_t *col, void *data)
{
if ( !data ) error("Error: the --merge-logic option cannot be used with FILTER (yet?)\n");
char *tmp;
int pos = strtol(tab->cols[col->icol], &tmp, 10);
if ( tmp==tab->cols[col->icol] )
- error("Could not parse ~POS at %s:%"PRId64" .. [%s]\n",bcf_seqname(args->hdr,line),(int64_t)line->pos+1,tab->cols[col->icol]);
+ error("Could not parse -POS at %s:%"PRId64" .. [%s]\n",bcf_seqname(args->hdr,line),(int64_t)line->pos+1,tab->cols[col->icol]);
line->pos = pos - 1;
return 0;
}
if (kh_exist(hash, k)) free((char*)kh_key(hash, k));
kh_clear(str2int, hash);
}
+static const char *escape_string(const char *str, char needle[], char **rmme, size_t *len)
+{
+ kstring_t tmp = {0,0,0};
+ const char *bp = str, *ep = str;
+ while ( *ep )
+ {
+ int i = 0;
+ while ( needle[i] && needle[i]!=*ep ) i++;
+ if ( !needle[i] ) { ep++; continue; }
+ kputsn(bp,ep-bp,&tmp);
+ ksprintf(&tmp,"%%%X",*ep);
+ bp = ++ep;
+ }
+ if ( !tmp.l )
+ {
+ *len = strlen(str);
+ return str;
+ }
+ kputs(bp,&tmp);
+ *len = tmp.l;
+ *rmme = tmp.s;
+ return tmp.s;
+}
static int setter_info_str(args_t *args, bcf1_t *line, annot_col_t *col, void *data)
{
if ( (col->replace & REPLACE_MISSING) && col->number!=BCF_VL_A && col->number!=BCF_VL_R )
if ( col->replace & SET_OR_APPEND ) col->merge_method=MM_UNIQUE;
annot_line_t *tab = (annot_line_t*) data;
+ const char *escaped = NULL;
+ char *rmme = NULL;
- int len = 0;
+ size_t len = 0;
if ( tab )
{
- len = strlen(tab->cols[col->icol]);
- if ( !len ) return 0;
- if ( len==1 && tab->cols[col->icol][0]=='.' && col->merge_method!=MM_APPEND_MISSING && !(col->replace & CARRY_OVER_MISSING) ) return 1;
+ char *str = tab->cols[col->icol];
+ if ( !str || !*str ) return 0;
+ if ( !str[1] && str[0]=='.' && col->merge_method!=MM_APPEND_MISSING && !(col->replace & CARRY_OVER_MISSING) ) return 1;
+ char needle[] = {';','=',0};
+ escaped = escape_string(tab->cols[col->icol],needle,&rmme,&len);
}
if ( col->merge_method!=MM_FIRST )
if ( col->merge_method==MM_UNIQUE )
{
if ( !col->mm_str_hash ) col->mm_str_hash = (khash_t(str2int)*)khash_str2int_init();
- if ( khash_str2int_has_key(col->mm_str_hash, tab->cols[col->icol]) ) return 1;
- khash_str2int_inc(col->mm_str_hash, strdup(tab->cols[col->icol]));
+ if ( khash_str2int_has_key(col->mm_str_hash, escaped) )
+ {
+ free(rmme);
+ return 1;
+ }
+ khash_str2int_inc(col->mm_str_hash, strdup(escaped));
}
if ( (col->replace & SET_OR_APPEND) && !col->mm_kstr.l )
}
if ( col->mm_kstr.l ) kputc(',',&col->mm_kstr);
- kputs(tab->cols[col->icol], &col->mm_kstr);
+ kputs(escaped, &col->mm_kstr);
+ free(rmme);
return 1;
}
-
if ( col->mm_kstr.l )
{
hts_expand(char,col->mm_kstr.l+1,args->mtmps,args->tmps);
memcpy(args->tmps,col->mm_kstr.s,col->mm_kstr.l+1);
}
else
+ {
+ free(rmme);
return 0;
+ }
// flush the line
if ( col->merge_method==MM_UNIQUE )
{
assert(tab);
hts_expand(char,len+1,args->mtmps,args->tmps);
- memcpy(args->tmps,tab->cols[col->icol],len+1);
-
+ memcpy(args->tmps,escaped,len+1);
if ( col->number==BCF_VL_A || col->number==BCF_VL_R )
return setter_ARinfo_string(args,line,col,tab->nals,tab->als);
}
-
- return bcf_update_info_string(args->hdr_out,line,col->hdr_key_dst,args->tmps);
+ int ret = bcf_update_info_string(args->hdr_out,line,col->hdr_key_dst,args->tmps);
+ free(rmme);
+ return ret;
}
static int vcf_setter_info_str(args_t *args, bcf1_t *line, annot_col_t *col, void *data)
{
if ( col->icol+args->nsmpl_annot > tab->ncols )
error("Incorrect number of values for %s at %s:%"PRId64"\n",col->hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ char needle[] = {':',0};
int ismpl;
for (ismpl=0; ismpl<args->nsmpl_annot; ismpl++)
- args->tmpp[ismpl] = tab->cols[col->icol + ismpl];
-
- return core_setter_format_str(args,line,col,args->tmpp);
+ {
+ size_t len;
+ char *rmme = NULL;
+ const char *str = escape_string(tab->cols[col->icol + ismpl],needle,&rmme,&len);
+ args->tmpp[ismpl] = rmme ? rmme : strdup(str);
+ }
+ int ret = core_setter_format_str(args,line,col,args->tmpp);
+ for (ismpl=0; ismpl<args->nsmpl_annot; ismpl++) free(args->tmpp[ismpl]);
+ return ret;
}
static int determine_ploidy(int nals, int *vals, int nvals1, uint8_t *smpl, int nsmpl)
{
kputsn(ss, se-ss, &str);
if ( !str.s[0] || !strcasecmp("-",str.s) ) ;
else if ( !strcasecmp("CHROM",str.s) ) args->chr_idx = icol;
- else if ( !strcasecmp("POS",str.s) ) args->beg_idx = icol;
+ else if ( !strcasecmp("POS",str.s) )
+ {
+ if ( replace==REPLACE_NON_MISSING && !args->tgts_is_vcf )
+ {
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = replace;
+ col->setter = setter_pos;
+ col->hdr_key_src = strdup(str.s);
+ col->hdr_key_dst = strdup(str.s);
+ args->match_end = icol;
+ }
+ else
+ args->beg_idx = icol;
+ }
else if ( !strcasecmp("FROM",str.s) || !strcasecmp("BEG",str.s) ) args->beg_idx = icol;
else if ( !strcasecmp("TO",str.s) || !strcasecmp("END",str.s) ) args->end_idx = icol;
else if ( !strcasecmp("REF",str.s) )
col->hdr_key_dst = strdup(str.s);
args->match_end = icol;
}
- else if ( !strcasecmp("~POS",str.s) && !args->tgts_is_vcf )
+ else if ( !strcasecmp("~POS",str.s) )
+ {
+ error("Error: the use of ~POS has been deprecated, use -POS to transfer the column POS.\n");
+ }
+ else if ( str.s[0]=='~' )
{
- if ( args->tgts_is_vcf ) error("Error: cannot use ~POS, position can be replaced only from a tab-delimited file\n");
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = MATCH_VALUE;
+ col->setter = NULL;
+ col->hdr_key_src = strdup(str.s+1);
+ }
+ else if ( !strcasecmp("-POS",str.s) && !args->tgts_is_vcf )
+ {
+ if ( args->tgts_is_vcf ) error("Error: cannot use -POS, position can be replaced only from a tab-delimited file\n");
args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
annot_col_t *col = &args->cols[args->ncols-1];
memset(col,0,sizeof(*col));
if ( bcf_hdr_id2type(args->tgts_hdr,BCF_HL_INFO,hdr_id)!=BCF_HT_STR )
error("Only Type=String tags can be used to annotate the ID column\n");
}
- else if ( (ptr=strstr(str.s,":=")) && !args->targets_fname )
+ else if ( (ptr=strstr(str.s,":=")) && (!args->targets_fname || !strncasecmp(ptr+2,"./",2)) )
{
*ptr = 0;
- rename_annots_push(args,ptr+2,str.s);
+ if ( !strncasecmp(str.s,"INFO/",5) && (!strcasecmp(ptr+2,"FILTER") || !strcasecmp(ptr+2,"./FILTER")) )
+ {
+ // -a not present and transferring filter, needs to be a local transfer
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = replace;
+ col->setter = vcf_setter_info_str;
+ col->getter = vcf_getter_filter2str_local;
+ col->hdr_key_src = strdup(ptr+2);
+ col->hdr_key_dst = strdup(str.s+5);
+ tmp.l = 0;
+ ksprintf(&tmp,"##INFO=<ID=%s,Number=1,Type=String,Description=\"Transferred FILTER column\">",col->hdr_key_dst);
+ bcf_hdr_append(args->hdr_out, tmp.s);
+ if (bcf_hdr_sync(args->hdr_out) < 0) error_errno("[%s] Failed to update header", __func__);
+ int hdr_id = bcf_hdr_id2int(args->hdr_out, BCF_DT_ID, col->hdr_key_dst);
+ col->number = bcf_hdr_id2length(args->hdr_out,BCF_HL_INFO,hdr_id);
+ }
+ else
+ rename_annots_push(args,ptr+2,str.s);
*ptr = ':';
}
else if ( !strcasecmp("FILTER",str.s) )
" (the annotation type is modified to \"Number=.\" and allele ordering is disregarded)\n");
fprintf(bcftools_stderr,"Warning: the =INFO/TAG feature modifies the annotation to \"Number=.\" and disregards allele ordering\n");
}
+
+ args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
+ annot_col_t *col = &args->cols[args->ncols-1];
+ memset(col,0,sizeof(*col));
+ col->icol = icol;
+ col->replace = replace;
+
int explicit_src_info = 0;
int explicit_dst_info = 0;
char *key_dst;
key_src[-2] = ':';
error("Did you mean \"FMT/%s\" rather than \"%s\"?\n",str.s,str.s);
}
+ else if ( !strcasecmp("FILTER",key_src) && args->tgts_is_vcf )
+ {
+ col->getter = vcf_getter_filter2str;
+ }
}
else
key_src = key_dst;
- args->ncols++; args->cols = (annot_col_t*) realloc(args->cols,sizeof(annot_col_t)*args->ncols);
- annot_col_t *col = &args->cols[args->ncols-1];
- memset(col,0,sizeof(*col));
- col->icol = icol;
- col->replace = replace;
col->hdr_key_src = strdup(key_src);
col->hdr_key_dst = strdup(key_dst);
while ( *ptr && isspace(*ptr) ) ptr++;
if ( !*ptr ) { *rmme = ' '; error("Could not parse: %s\n", args->rename_annots_map[i]); }
if ( rename_annots_core(args, args->rename_annots_map[i], ptr) < 0 )
- error("Could not parse \"%s %s\", expected INFO, FORMAT, or FILTER prefix\n",args->rename_annots_map[i],ptr);
+ error("Cannot rename \"%s\" to \"%s\"\n",args->rename_annots_map[i],ptr);
}
}
static void rename_annots_push(args_t *args, char *src, char *dst)
ksprintf(&str,"%s %s",src,dst);
args->rename_annots_map[ args->rename_annots_nmap - 1 ] = str.s;
}
+static void init_filters(args_t *args)
+{
+ // Check if the -i/-e expressions contain external values that should be determined
+ // on the fly from the annotation file. The expressions can be given as
+ // TAG={NAME}
+ // TAG={str:NAME}
+ // TAG={int:NAME}
+ // TAG={float:NAME}
+ kstring_t str = {0,0,0};
+ char *src = strdup(args->filter_str);
+ int len = 0;
+ while (1)
+ {
+ char *beg = strchr(src+len,'{');
+ if ( !beg ) break;
+
+ // check if "{" appears inside quotes, in such case do not modify
+ char skip = 0;
+ char *tmp = src;
+ while ( tmp<beg )
+ {
+ if ( tmp[0]!='"' && tmp[0]!='\'' ) { tmp++; continue; }
+
+ // quote character found
+ int quote = tmp[0];
+ tmp++;
+ while ( *tmp && tmp[0]!=quote ) tmp++;
+ if ( !*tmp ) error("Could not parse the expression: %s\n",args->filter_str); // unbalanced quotation; todo: check for escape char
+ len = tmp - src + 1;
+ skip = 1;
+ }
+ if ( skip ) continue;
+
+ char *end = ++beg;
+ while ( *end && *end!='}' ) end++;
+ if ( !*end ) error("Could not parse the expression: %s\n",args->filter_str);
+ *end = 0;
+
+ // explicit typing?
+ int type = -1;
+ tmp = beg;
+ while ( *tmp && *tmp!=':' ) tmp++;
+ if ( *tmp )
+ {
+ *tmp = 0;
+ if ( !strcasecmp(beg,"str") ) type = BCF_HT_STR;
+ else if ( !strcasecmp(beg,"int") ) type = BCF_HT_INT;
+ else if ( !strcasecmp(beg,"float") ) type = BCF_HT_REAL;
+ }
+ args->n_ext++;
+ args->ext = (ext_t*)realloc(args->ext,sizeof(*args->ext)*args->n_ext);
+ ext_t *ext = &args->ext[args->n_ext-1];
+ ext->ht_type = type;
+ ext->name = strdup(beg);
+ if ( beg-1 > src ) kputsn(src,beg-1-src,&str);
+ if ( type==-1 ) kputs("{}",&str);
+ else if ( type==BCF_HT_STR ) kputs("{str}",&str);
+ else if ( type==BCF_HT_INT ) kputs("{int}",&str);
+ else if ( type==BCF_HT_REAL ) kputs("{float}",&str);
+ len = str.l;
+ kputs(end+1,&str);
+ free(src);
+ src = strdup(str.s);
+ str.l = 0;
+ }
+ args->filter = filter_init(args->hdr, src);
+ free(src);
+ free(str.s);
+
+ int i,j,n_ext;
+ const int *ext_type = filter_ext_types(args->filter, &n_ext);
+ if ( n_ext != args->n_ext )
+ error("Failed to parse the expression, unexpected number of dynamic variables (%d vs %d): %s\n",n_ext,args->n_ext,args->filter_str);
+
+ if ( !args->n_ext ) return;
+
+ if ( !args->tgts )
+ error("Error: dynamic variables in -i/-e expressions can be currently used only with tab-delimited file, not with VCF (todo)\n");
+
+ // contains external values
+ args->ext_ptr = malloc(sizeof(*args->ext_ptr)*args->n_ext);
+ for (i=0; i<args->n_ext; i++) args->ext[i].ht_type = ext_type[i];
+ args->filter_ext = args->filter;
+ args->filter = NULL;
+
+ // set the column idx
+ if ( args->ncols )
+ {
+ for (i=0; i<args->n_ext; i++)
+ {
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( strcmp(args->ext[i].name,args->cols[j].hdr_key_src) ) continue;
+ args->ext[i].icol = args->cols[j].icol;
+ break;
+ }
+ if ( j==args->ncols ) error("No such column: %s\n",args->ext[i].name);
+ }
+ }
+}
static void init_data(args_t *args)
{
args->vcmp = vcmp_init();
if ( args->filter_str )
- args->filter = filter_init(args->hdr, args->filter_str);
+ init_filters(args);
if ( args->mark_sites )
{
if ( args->n_threads )
hts_set_opt(args->out_fh, HTS_OPT_THREAD_POOL, args->files->p);
if ( bcf_hdr_write(args->out_fh, args->hdr_out)!=0 ) error("[%s] Error: failed to write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr,args->output_fname,
+ &args->index_fn, args->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
}
}
static void destroy_data(args_t *args)
{
int i;
+ for (i=0; i<args->n_ext; i++)
+ {
+ free(args->ext[i].name);
+ if ( args->ext[i].ht_type!=BCF_HT_STR ) continue;
+ }
+ free(args->ext_ptr);
+ free(args->ext);
for (i=0; i<args->nrm; i++) free(args->rm[i].key);
free(args->rm);
if ( args->hdr_out ) bcf_hdr_destroy(args->hdr_out);
free(args->dst_smpl_pld);
if ( args->set_ids )
convert_destroy(args->set_ids);
- if ( args->filter )
- filter_destroy(args->filter);
+ if ( args->filter ) filter_destroy(args->filter);
+ if ( args->filter_ext ) filter_destroy(args->filter_ext);
if (args->out_fh)
{
if ( args->write_index )
}
else i++;
}
- if ( args->ref_idx==-1 && args->nalines ) return;
+ if ( !args->filter_ext && args->ref_idx==-1 && args->nalines ) return;
while ( !bcf_sr_regions_overlap(args->tgts, bcf_seqname(args->hdr,line), start_pos,end_pos) )
{
tmp->start = args->tgts->start;
tmp->end = args->tgts->end;
parse_annot_line(args, args->tgts->line.s, tmp);
- if ( args->ref_idx != -1 )
+ if ( args->filter_ext || args->ref_idx != -1 )
{
int iseq = args->tgts->iseq;
if ( bcf_sr_regions_next(args->tgts)<0 || args->tgts->iseq!=iseq ) break;
}
return 0;
}
-static void annotate(args_t *args, bcf1_t *line)
+static int annotate_from_regidx(args_t *args, bcf1_t *line)
{
- int i, j;
- for (i=0; i<args->nrm; i++)
- args->rm[i].handler(args, line, &args->rm[i]);
-
+ int j;
int has_overlap = 0;
- if ( args->tgt_idx )
+
+ for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
+ if ( regidx_overlap(args->tgt_idx, bcf_seqname(args->hdr,line),line->pos,line->pos+line->rlen-1, args->tgt_itr) )
{
- for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
- if ( regidx_overlap(args->tgt_idx, bcf_seqname(args->hdr,line),line->pos,line->pos+line->rlen-1, args->tgt_itr) )
+ hts_pos_t vcf_end = line->pos + line->rlen - 1;
+ while ( regitr_overlap(args->tgt_itr) )
{
- hts_pos_t vcf_end = line->pos + line->rlen - 1;
- while ( regitr_overlap(args->tgt_itr) )
- {
- annot_line_t *tmp = &args->alines[0];
- tmp->rid = line->rid;
- tmp->start = args->tgt_itr->beg;
- tmp->end = args->tgt_itr->end;
+ annot_line_t *tmp = &args->alines[0];
+ tmp->rid = line->rid;
+ tmp->start = args->tgt_itr->beg;
+ tmp->end = args->tgt_itr->end;
- // Check min overlap
- int len_ann = tmp->end - tmp->start + 1;
- int len_vcf = line->rlen;
- int isec = (tmp->end < vcf_end ? tmp->end : vcf_end) - (tmp->start > line->pos ? tmp->start : line->pos) + 1;
- assert( isec > 0 );
- if ( args->min_overlap_ann && args->min_overlap_ann > (float)isec/len_ann ) continue;
- if ( args->min_overlap_vcf && args->min_overlap_vcf > (float)isec/len_vcf ) continue;
+ // Check min overlap
+ int len_ann = tmp->end - tmp->start + 1;
+ int len_vcf = line->rlen;
+ int isec = (tmp->end < vcf_end ? tmp->end : vcf_end) - (tmp->start > line->pos ? tmp->start : line->pos) + 1;
+ assert( isec > 0 );
+ if ( args->min_overlap_ann && args->min_overlap_ann > (float)isec/len_ann ) continue;
+ if ( args->min_overlap_vcf && args->min_overlap_vcf > (float)isec/len_vcf ) continue;
- parse_annot_line(args, regitr_payload(args->tgt_itr,char*), tmp);
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].done==1 ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],tmp);
- if ( ret < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- if ( ret==0 )
- args->cols[j].done = 1;
- has_overlap = 1;
- }
+ parse_annot_line(args, regitr_payload(args->tgt_itr,char*), tmp);
+
+ // If a plain BED file is provided and we are asked to just mark overlapping sites, there are
+ // no additional columns. Not sure if there can be any side effects for ill-formatted BED files
+ // with variable number of columns
+ if ( !args->ncols && args->mark_sites ) has_overlap = 1;
+
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],tmp);
+ if ( ret < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ if ( ret==0 )
+ args->cols[j].done = 1;
+ has_overlap = 1;
}
}
- for (j=0; j<args->ncols; j++)
+ }
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
+ if ( !args->cols[j].setter ) continue;
+ if ( args->cols[j].setter(args,line,&args->cols[j],NULL) < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ }
+ return has_overlap;
+}
+static int pass_filter_test_ext(args_t *args, bcf1_t *line, annot_line_t *ann)
+{
+ char *tmp;
+ int i;
+ for (i=0; i<args->n_ext; i++)
+ {
+ int j = args->ext[i].icol;
+ if ( args->ext[i].ht_type==BCF_HT_STR ) args->ext_ptr[i] = args->ext[i].s = ann->cols[j];
+ else if ( args->ext[i].ht_type==BCF_HT_INT )
{
- if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
- if ( !args->cols[j].setter ) continue;
- if ( args->cols[j].setter(args,line,&args->cols[j],NULL) < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ args->ext[i].i = strtol(ann->cols[j],&tmp,10);
+ if ( *tmp )
+ {
+ if ( strcmp(".",ann->cols[j]) ) error("Error: could not parse the annotation file, expected an integer, found \"%s\"\n",ann->cols[j]);
+ args->ext_ptr[i] = NULL;
+ }
+ else
+ args->ext_ptr[i] = &args->ext[i].i;
+ }
+ else if ( args->ext[i].ht_type==BCF_HT_REAL )
+ {
+ args->ext[i].f = strtod(ann->cols[j],&tmp);
+ if ( *tmp )
+ {
+ if ( strcmp(".",ann->cols[j]) ) error("Error: could not parse the annotation file, expected a float, found \"%s\"\n",ann->cols[j]);
+ args->ext_ptr[i] = NULL;
+ }
+ else
+ args->ext_ptr[i] = &args->ext[i].f;
}
}
- else if ( args->tgts )
- {
- // Buffer annotation lines. When multiple ALT alleles are present in the annotation file, at least one
- // must match some of the VCF alleles. If the append-missing mode is set (and REF+ALT is requested), the
- // buffered lines will annotate the VCF respecting the order in ALT and when no matching line is found
- // for an ALT, missing value is appended instead.
- int end_pos = line->pos + line->rlen - 1;
- buffer_annot_lines(args, line, line->pos, end_pos);
+ int pass = filter_test_ext(args->filter_ext,line,NULL,(const void**)args->ext_ptr);
+ if ( args->filter_logic==FLT_EXCLUDE ) pass = pass ? 0 : 1;
+ return pass;
+}
+static int annotate_from_tab(args_t *args, bcf1_t *line)
+{
+ int i,j;
+ int has_overlap = 0;
+
+ // Buffer annotation lines. When multiple ALT alleles are present in the annotation file, at least one
+ // must match some of the VCF alleles. If the append-missing mode is set (and REF+ALT is requested), the
+ // buffered lines will annotate the VCF respecting the order in ALT and when no matching line is found
+ // for an ALT, missing value is appended instead.
+ int end_pos = line->pos + line->rlen - 1;
+ buffer_annot_lines(args, line, line->pos, end_pos);
- args->nsrt_alines = 0;
- hts_expand(uint32_t,args->nalines,args->msrt_alines,args->srt_alines);
- if ( args->nalines >= 0xffff || line->n_allele >= 0xffff )
- error("Error: too many alleles or annotation lines in the buffer at %s:%"PRId64" (todo:skip?)\n",bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ args->nsrt_alines = 0;
+ hts_expand(uint32_t,args->nalines,args->msrt_alines,args->srt_alines);
+ if ( args->nalines >= 0xffff || line->n_allele >= 0xffff )
+ error("Error: too many alleles or annotation lines in the buffer at %s:%"PRId64" (todo:skip?)\n",bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- kstring_t match_end = {0,0,0};
- if ( args->match_end>=0 && bcf_get_info_int32(args->hdr,line,"END",&args->tmpi,&args->mtmpi)==1 )
- kputw(args->tmpi[0],&match_end);
+ kstring_t match_end = {0,0,0};
+ if ( args->match_end>=0 && bcf_get_info_int32(args->hdr,line,"END",&args->tmpi,&args->mtmpi)==1 )
+ kputw(args->tmpi[0],&match_end);
- // Find matching lines
- for (i=0; i<args->nalines; i++)
+ // Find matching lines
+ for (i=0; i<args->nalines; i++)
+ {
+ if ( line->pos > args->alines[i].end || end_pos < args->alines[i].start ) continue;
+ if ( args->ref_idx != -1 ) // REF+ALT matching requested
{
- if ( line->pos > args->alines[i].end || end_pos < args->alines[i].start ) continue;
- if ( args->ref_idx != -1 ) // REF+ALT matching requested
+ if ( line->pos!=args->alines[i].start || vcmp_set_ref(args->vcmp, line->d.allele[0], args->alines[i].als[0]) < 0 ) continue; // refs are not compatible
+ for (j=1; j<args->alines[i].nals; j++)
{
- if ( line->pos!=args->alines[i].start || vcmp_set_ref(args->vcmp, line->d.allele[0], args->alines[i].als[0]) < 0 ) continue; // refs are not compatible
- for (j=1; j<args->alines[i].nals; j++)
+ int ialt;
+ if ( line->n_allele==1 && args->alines[i].als[j][0]=='.' && args->alines[i].als[j][1]==0 ) // match: no ALT allele in VCF and annot file has "."
+ ialt = 0;
+ else
{
- int ialt;
- if ( line->n_allele==1 && args->alines[i].als[j][0]=='.' && args->alines[i].als[j][1]==0 ) // match: no ALT allele in VCF and annot file has "."
- ialt = 0;
- else
- {
- ialt = vcmp_find_allele(args->vcmp, line->d.allele+1, line->n_allele - 1, args->alines[i].als[j]);
- if ( ialt < 0 ) continue;
- ialt++;
- }
- if ( args->match_id>=0 && !strstr_match(line->d.id,args->alines[i].cols[args->match_id]) ) continue;
- if ( args->match_end>=0 && match_end.l && strcmp(match_end.s,args->alines[i].cols[args->match_end]) ) continue;
- args->srt_alines[args->nsrt_alines++] = (ialt<<16) | i;
- has_overlap = 1;
- break;
+ ialt = vcmp_find_allele(args->vcmp, line->d.allele+1, line->n_allele - 1, args->alines[i].als[j]);
+ if ( ialt < 0 ) continue;
+ ialt++;
}
+ if ( args->match_id>=0 && !strstr_match(line->d.id,args->alines[i].cols[args->match_id]) ) continue;
+ if ( args->match_end>=0 && match_end.l && strcmp(match_end.s,args->alines[i].cols[args->match_end]) ) continue;
+ if ( args->filter_ext && !pass_filter_test_ext(args,line,&args->alines[i]) ) continue;
+ args->srt_alines[args->nsrt_alines++] = (ialt<<16) | i;
+ has_overlap = 1;
+ break;
}
- else // overlap, REF+ALT matching not requested
+ }
+ else if ( args->filter_ext )
+ {
+ if ( pass_filter_test_ext(args,line,&args->alines[i]) )
{
args->srt_alines[args->nsrt_alines++] = (0xffff<<16) | i;
has_overlap = 1;
}
}
+ else // overlap, REF+ALT matching not requested
+ {
+ args->srt_alines[args->nsrt_alines++] = (0xffff<<16) | i;
+ has_overlap = 1;
+ }
+ }
- free(match_end.s);
+ free(match_end.s);
+ if ( !has_overlap && args->filter_ext && !args->keep_sites ) return has_overlap;
- // Sort lines if needed
+ // Sort lines if needed
+ if ( args->has_append_mode )
+ {
+ // insertion sort by VCF ALT index (top bits) and alines index (low bits)
+ uint32_t tmp;
+ for (i=1; i<args->nsrt_alines; i++)
+ for (j=i; j>0 && args->srt_alines[j] < args->srt_alines[j-1]; j--)
+ tmp = args->srt_alines[j], args->srt_alines[j] = args->srt_alines[j-1], args->srt_alines[j-1] = tmp;
+ }
+ // Annotate
+ for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
+ int ialt_exp = 1;
+ for (i=0; i<args->nsrt_alines; i++)
+ {
+ int ialt = args->srt_alines[i] >> 16;
+ int ilin = args->srt_alines[i] & 0xffff;
if ( args->has_append_mode )
{
- // insertion sort by VCF ALT index (top bits) and alines index (low bits)
- uint32_t tmp;
- for (i=1; i<args->nsrt_alines; i++)
- for (j=i; j>0 && args->srt_alines[j] < args->srt_alines[j-1]; j--)
- tmp = args->srt_alines[j], args->srt_alines[j] = args->srt_alines[j-1], args->srt_alines[j-1] = tmp;
- }
- // Annotate
- for (j=0; j<args->ncols; j++) args->cols[j].done = 0;
- int ialt_exp = 1;
- for (i=0; i<args->nsrt_alines; i++)
- {
- int ialt = args->srt_alines[i] >> 16;
- int ilin = args->srt_alines[i] & 0xffff;
- if ( args->has_append_mode )
+ if ( ialt_exp > ialt ) continue; // multiple annotation lines for the same position
+ if ( ialt_exp < ialt )
{
- if ( ialt_exp > ialt ) continue; // multiple annotation lines for the same position
- if ( ialt_exp < ialt )
- {
- // REF+ALT matching requested, append-missing mode: insert "." if no annotation line was found for the ALT
- while ( ialt_exp++ < ialt )
- {
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].merge_method != MM_APPEND_MISSING ) continue;
- if ( args->cols[j].done==1 ) continue;
- if ( !args->cols[j].setter ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],args->aline_missing);
- if ( ret < 0 )
- error("fixme: Could not set missing %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- if ( ret==0 )
- args->cols[j].done = 1;
- }
- }
- }
- }
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].done==1 ) continue;
- if ( !args->cols[j].setter ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],&args->alines[ilin]);
- if ( ret < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- if ( ret==0 )
- args->cols[j].done = 1;
- }
- ialt_exp = ialt + 1;
- }
- if ( args->nsrt_alines )
- {
- // In the append-missing mode fill missing values to all trailing ALTs, but only if at least one
- // record was found. Otherwise leave the row will be left without annotation.
- if ( args->has_append_mode && ialt_exp < line->n_allele )
- {
- while ( ialt_exp++ < line->n_allele )
+ // REF+ALT matching requested, append-missing mode: insert "." if no annotation line was found for the ALT
+ while ( ialt_exp++ < ialt )
{
for (j=0; j<args->ncols; j++)
{
}
}
}
- // Flush
- for (j=0; j<args->ncols; j++)
- {
- if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
- if ( !args->cols[j].setter ) continue;
- int ret = args->cols[j].setter(args,line,&args->cols[j],NULL);
- if ( ret < 0 )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
- }
}
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 ) continue;
+ if ( !args->cols[j].setter ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],&args->alines[ilin]);
+ if ( ret < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ if ( ret==0 )
+ args->cols[j].done = 1;
+ }
+ ialt_exp = ialt + 1;
}
- else if ( args->files->nreaders == 2 )
+ if ( args->nsrt_alines )
{
- if ( bcf_sr_has_line(args->files,1) )
+ // In the append-missing mode fill missing values to all trailing ALTs, but only if at least one
+ // record was found. Otherwise leave the row will be left without annotation.
+ if ( args->has_append_mode && ialt_exp < line->n_allele )
{
- bcf1_t *aline = bcf_sr_get_line(args->files,1);
- for (j=0; j<args->ncols; j++)
+ while ( ialt_exp++ < line->n_allele )
{
- if ( !args->cols[j].setter ) continue;
- if ( args->cols[j].setter(args,line,&args->cols[j],aline) )
- error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].merge_method != MM_APPEND_MISSING ) continue;
+ if ( args->cols[j].done==1 ) continue;
+ if ( !args->cols[j].setter ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],args->aline_missing);
+ if ( ret < 0 )
+ error("fixme: Could not set missing %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ if ( ret==0 )
+ args->cols[j].done = 1;
+ }
}
-
- has_overlap = 1;
}
+ // Flush
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( args->cols[j].done==1 || args->cols[j].merge_method == MM_FIRST ) continue;
+ if ( !args->cols[j].setter ) continue;
+ int ret = args->cols[j].setter(args,line,&args->cols[j],NULL);
+ if ( ret < 0 )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ }
+ }
+ return has_overlap;
+}
+static int annotate_from_vcf(args_t *args, bcf1_t *line)
+{
+ if ( !bcf_sr_has_line(args->files,1) ) return 0;
+ int j;
+ bcf1_t *aline = bcf_sr_get_line(args->files,1);
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( !args->cols[j].setter ) continue;
+ if ( args->cols[j].setter(args,line,&args->cols[j],aline) )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
}
+ return 1;
+}
+static int annotate_from_self(args_t *args, bcf1_t *line)
+{
+ int j;
+ for (j=0; j<args->ncols; j++)
+ {
+ if ( !args->cols[j].setter ) continue;
+ if ( args->cols[j].setter(args,line,&args->cols[j],NULL) )
+ error("fixme: Could not set %s at %s:%"PRId64"\n", args->cols[j].hdr_key_src,bcf_seqname(args->hdr,line),(int64_t) line->pos+1);
+ }
+ return 0;
+}
+static int annotate_line(args_t *args, bcf1_t *line)
+{
+ args->current_rec = line;
+
+ int i;
+ for (i=0; i<args->nrm; i++)
+ args->rm[i].handler(args, line, &args->rm[i]);
+
+ int has_overlap = 0;
+ if ( args->tgt_idx )
+ has_overlap = annotate_from_regidx(args,line);
+
+ else if ( args->tgts )
+ has_overlap = annotate_from_tab(args,line);
+
+ else if ( args->files->nreaders == 2 )
+ has_overlap = annotate_from_vcf(args,line);
+
+ else if ( args->ncols )
+ has_overlap = annotate_from_self(args,line);
+
if ( args->set_ids )
{
args->tmpks.l = 0;
else
bcf_update_info_flag(args->hdr_out,line,args->mark_sites,NULL,has_overlap?0:1);
}
+
+ return has_overlap;
}
static void usage(args_t *args)
fprintf(bcftools_stderr, " --single-overlaps Keep memory low by avoiding complexities arising from handling multiple overlapping intervals\n");
fprintf(bcftools_stderr, " -x, --remove LIST List of annotations (e.g. ID,INFO/DP,FORMAT/DP,FILTER) to remove (or keep with \"^\" prefix). See man page for details\n");
fprintf(bcftools_stderr, " --threads INT Number of extra output compression threads [0]\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Examples:\n");
fprintf(bcftools_stderr, " http://samtools.github.io/bcftools/howtos/annotate.html\n");
{"min-overlap",required_argument,NULL,12},
{"no-version",no_argument,NULL,8},
{"force",no_argument,NULL,'f'},
- {"write-index",no_argument,NULL,13},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "h:H:?o:O:r:R:a:x:c:C:i:e:S:s:I:m:kl:f",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h:H:?o:O:r:R:a:x:c:C:i:e:S:s:I:m:kl:fW::",loptions,NULL)) >= 0)
{
switch (c) {
case 'f': args->force = 1; break;
case 10 : args->single_overlaps = 1; break;
case 11 : args->rename_annots = optarg; break;
case 12 : args->min_overlap_str = optarg; break;
- case 13 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
}
continue;
}
}
- annotate(args, line);
+ int keep = annotate_line(args, line);
+ if ( args->filter_ext && !args->keep_sites && !keep ) continue;
if ( bcf_write1(args->out_fh, args->hdr_out, line)!=0 ) error("[%s] Error: failed to write to %s\n", __func__,args->output_fname);
}
destroy_data(args);
/* The MIT License
- Copyright (c) 2016-2022 Genome Research Ltd.
+ Copyright (c) 2016-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/vcf.h>
#include <htslib/vcfutils.h>
#include <htslib/hts_os.h>
+#include <htslib/kbitset.h>
#include "bcftools.h"
#include "vcfbuf.h"
#include "rbuf.h"
{
bcf1_t *rec;
double af;
- int af_set:1, filter:1, idx:30;
+ unsigned int af_set:1, filter:1, idx:30;
}
vcfrec_t;
}
prune_t;
+
+#define MARK_OVERLAP 1
+#define MARK_DUP 2
+#define MARK_EXPR 3
+
+#define MARK_MISSING_SCALAR 0 // actual value to use
+#define MARK_MISSING_MAX_DP 1 // max overlap_t.value scaled by INFO/DP
+
+// temporary internal structure for iterative overlap removal by mark_t.expr
typedef struct
{
- int active;
+ double value; // the sort value
+ int rmme, idx; // mark for removal, index in vcfbuf_t.rbuf
+ int dp; // with MARK_MISSING_MAX_DP, INFO/DP is used extrapolate missing QUAL
+ kbitset_t *bset; // mark which records it overlaps with, given as 0-based indexes to vcfbuf_t.rbuf
+ bcf1_t *rec;
}
-rmdup_t;
-
+overlap_t;
typedef struct
{
- int active, rid, end;
+ // modes
+ int mode;
+ char *expr;
+
+ // sites marked according to expr, returned to the caller via vcfbuf_get()
+ rbuf_t rbuf;
+ uint8_t *mark;
+ int last;
+
+ // MARK_OVERLAP
+ int overlap_rid, overlap_end;
+
+ // MARK_EXPR
+ int nbuf;
+ overlap_t *buf, **buf_ptr;
+ int missing_expr; // the value to use when min(QUAL) encounters a missing value
+ float missing_value; // the default missing value
+ float max_qual; // with MARK_MISSING_MAX_DP
+ int max_qual_dp; //
+ int ntmpi; // temporary int array and the allocated memory
+ int32_t *tmpi;
}
-overlap_t;
+mark_t;
struct _vcfbuf_t
{
- int win, dummy;
+ int win, // maximum number of sites in the buffer, either number of sites (<0) or bp (<0)
+ dummy; // the caller maintains the buffer via push/peek/flush
bcf_hdr_t *hdr;
vcfrec_t *vcf;
rbuf_t rbuf;
ld_t ld;
prune_t prune;
- overlap_t overlap;
- rmdup_t rmdup;
+ mark_t mark;
+ enum { clean, dirty } status;
};
vcfbuf_t *vcfbuf_init(bcf_hdr_t *hdr, int win)
vcfbuf_t *buf = (vcfbuf_t*) calloc(1,sizeof(vcfbuf_t));
buf->hdr = hdr;
buf->win = win;
- buf->overlap.rid = -1;
+ buf->status = clean;
+ buf->mark.overlap_rid = -1;
int i;
for (i=0; i<VCFBUF_LD_N; i++) buf->ld.max[i] = HUGE_VAL;
rbuf_init(&buf->rbuf, 0);
free(buf->prune.farr);
free(buf->prune.vrec);
free(buf->prune.ac);
+ free(buf->prune.af_tag);
free(buf->prune.idx);
+ free(buf->mark.mark);
+ free(buf->mark.expr);
+ for (i=0; i<buf->mark.nbuf; i++) kbs_destroy(buf->mark.buf[i].bset);
+ free(buf->mark.buf);
+ free(buf->mark.buf_ptr);
+ free(buf->mark.tmpi);
free(buf);
}
-void vcfbuf_set(vcfbuf_t *buf, vcfbuf_opt_t key, void *value)
+int vcfbuf_set(vcfbuf_t *buf, vcfbuf_opt_t key, ...)
{
- if ( key==LD_FILTER1 ) { buf->ld.filter1 = *((int*)value); return; }
- if ( key==LD_RAND_MISSING ) { buf->ld.rand_missing = *((int*)value); return; }
- if ( key==LD_MAX_R2 ) { buf->ld.max[VCFBUF_LD_IDX_R2] = *((double*)value); return; }
- if ( key==LD_MAX_LD ) { buf->ld.max[VCFBUF_LD_IDX_LD] = *((double*)value); return; }
- if ( key==LD_MAX_HD ) { buf->ld.max[VCFBUF_LD_IDX_HD] = *((double*)value); return; }
-
- if ( key==VCFBUF_DUMMY ) { buf->dummy = *((int*)value); return; }
- if ( key==VCFBUF_NSITES )
+ va_list args;
+ switch (key)
{
- buf->prune.max_sites = *((int*)value);
- if ( !buf->prune.mode ) buf->prune.mode = PRUNE_MODE_MAX_AF;
- return;
+ case LD_FILTER1:
+ va_start(args, key);
+ buf->ld.filter1 = va_arg(args,int);
+ va_end(args);
+ return 0;
+
+ case LD_RAND_MISSING:
+ va_start(args, key);
+ buf->ld.rand_missing = va_arg(args,int);
+ va_end(args);
+ return 0;
+
+ case LD_MAX_R2:
+ va_start(args, key);
+ buf->ld.max[VCFBUF_LD_IDX_R2] = va_arg(args,double);
+ va_end(args);
+ return 0;
+
+ case LD_MAX_LD:
+ va_start(args, key);
+ buf->ld.max[VCFBUF_LD_IDX_LD] = va_arg(args,double);
+ va_end(args);
+ return 0;
+
+ case LD_MAX_HD:
+ va_start(args, key);
+ buf->ld.max[VCFBUF_LD_IDX_HD] = va_arg(args,double);
+ va_end(args);
+ return 0;
+
+ case VCFBUF_DUMMY:
+ va_start(args, key);
+ buf->dummy = va_arg(args,int);
+ va_end(args);
+ return 0;
+
+ case PRUNE_NSITES:
+ va_start(args, key);
+ buf->prune.max_sites = va_arg(args,int);
+ if ( !buf->prune.mode ) buf->prune.mode = PRUNE_MODE_MAX_AF;
+ va_end(args);
+ return 0;
+
+ case PRUNE_NSITES_MODE:
+ va_start(args, key);
+ char *mode = va_arg(args,char*);
+ va_end(args);
+ if ( !strcasecmp(mode,"maxAF") ) buf->prune.mode = PRUNE_MODE_MAX_AF;
+ else if ( !strcasecmp(mode,"1st") ) buf->prune.mode = PRUNE_MODE_1ST;
+ else if ( !strcasecmp(mode,"rand") ) buf->prune.mode = PRUNE_MODE_RAND;
+ else error("The mode \"%s\" is not recognised\n",mode);
+ return 0;
+
+ case PRUNE_AF_TAG:
+ va_start(args, key);
+ buf->prune.af_tag = strdup(va_arg(args,char*));
+ va_end(args);
+ return 0;
+
+ case MARK:
+ va_start(args, key);
+ buf->mark.expr = strdup(va_arg(args,char*));
+ if ( !strcasecmp(buf->mark.expr,"overlap") ) buf->mark.mode = MARK_OVERLAP;
+ else if ( !strcasecmp(buf->mark.expr,"dup") ) buf->mark.mode = MARK_DUP;
+ else buf->mark.mode = MARK_EXPR;
+ va_end(args);
+ return 0;
+
+ case MARK_MISSING_EXPR:
+ va_start(args, key);
+ char *expr = va_arg(args,char*);
+ if ( !strcasecmp(expr,"0") )
+ {
+ buf->mark.missing_expr = MARK_MISSING_SCALAR;
+ buf->mark.missing_value = 0;
+ }
+ else if ( !strcasecmp(expr,"DP") )
+ {
+ if ( buf->mark.mode!=MARK_EXPR ) error("Only the combination of --mark 'min(QUAL)' with --missing DP is currently supported\n");
+ buf->mark.missing_expr = MARK_MISSING_MAX_DP;
+ }
+ else
+ error("todo: MARK_MISSING_EXPR=%s\n",expr);
+ va_end(args);
+ return 0;
}
- if ( key==VCFBUF_AF_TAG ) { buf->prune.af_tag = *((char**)value); return; }
- if ( key==VCFBUF_OVERLAP_WIN ) { buf->overlap.active = *((int*)value); return; }
- if ( key==VCFBUF_RMDUP) { buf->rmdup.active = *((int*)value); return; }
+ return 0;
+}
- if ( key==VCFBUF_NSITES_MODE )
- {
- char *mode = *((char**)value);
- if ( !strcasecmp(mode,"maxAF") ) buf->prune.mode = PRUNE_MODE_MAX_AF;
- else if ( !strcasecmp(mode,"1st") ) buf->prune.mode = PRUNE_MODE_1ST;
- else if ( !strcasecmp(mode,"rand") ) buf->prune.mode = PRUNE_MODE_RAND;
- else error("The mode \"%s\" is not recognised\n",mode);
- return;
- }
+void *vcfbuf_get(vcfbuf_t *buf, vcfbuf_opt_t key, ...)
+{
+ va_list args;
+ va_start(args, key);
+ if ( key==MARK )
+ return &buf->mark.last;
+ va_end(args);
+ return NULL;
}
int vcfbuf_nsites(vcfbuf_t *buf)
bcf1_t *vcfbuf_push(vcfbuf_t *buf, bcf1_t *rec)
{
- rbuf_expand0(&buf->rbuf, vcfrec_t, buf->rbuf.n+1, buf->vcf);
+ // make sure the caller is using the buffer correctly and calls vcfbuf_flush()
+ // before placing next vcfbuf_push() call
+ assert(buf->status!=dirty);
+ if ( !buf->dummy ) buf->status = dirty;
+ rbuf_expand0(&buf->rbuf, vcfrec_t, buf->rbuf.n+1, buf->vcf);
int i = rbuf_append(&buf->rbuf);
if ( !buf->vcf[i].rec ) buf->vcf[i].rec = bcf_init1();
bcf1_t *vcfbuf_peek(vcfbuf_t *buf, int idx)
{
+ buf->status = clean;
int i = rbuf_kth(&buf->rbuf, idx);
return i<0 ? NULL : buf->vcf[i].rec;
}
static void _prune_sites(vcfbuf_t *buf, int flush_all)
{
+
int nbuf = flush_all ? buf->rbuf.n : buf->rbuf.n - 1;
int nprune = nbuf - buf->prune.max_sites;
rbuf_remove_kth(&buf->rbuf, vcfrec_t, buf->prune.idx[i], buf->vcf);
}
-static int _rmdup_can_flush(vcfbuf_t *buf, int flush_all)
+static int mark_dup_can_flush_(vcfbuf_t *buf, int flush_all)
{
- if ( flush_all ) return 1;
+ int flush = flush_all;
+ mark_t *mark = &buf->mark;
+ if ( buf->status==dirty )
+ {
+ // a new site was just added by vcfbuf_push()
+ rbuf_expand0(&mark->rbuf, uint8_t, buf->rbuf.n, mark->mark);
+ int i = rbuf_append(&mark->rbuf);
+ mark->mark[i] = 0;
+
+ if ( buf->rbuf.n==1 ) goto flush;
- if ( buf->rbuf.n==1 ) return 0;
+ // there is at least one previous site, check if it's a duplicate
+ int k1 = rbuf_kth(&buf->rbuf, -1);
+ int k2 = rbuf_kth(&buf->rbuf, -2);
+ vcfrec_t *rec1 = &buf->vcf[k1];
+ vcfrec_t *rec2 = &buf->vcf[k2];
- int k1 = rbuf_kth(&buf->rbuf, -1);
- int k2 = rbuf_kth(&buf->rbuf, -2);
+ int is_dup = 1;
+ if ( rec1->rec->rid!=rec2->rec->rid ) is_dup = 0;
+ else if ( rec1->rec->pos!=rec2->rec->pos ) is_dup = 0;
- vcfrec_t *rec1 = &buf->vcf[k1];
- vcfrec_t *rec2 = &buf->vcf[k2];
+ if ( is_dup )
+ {
+ // it is, mark the last two sites as duplicates
+ int k1 = rbuf_kth(&mark->rbuf, -1);
+ int k2 = rbuf_kth(&mark->rbuf, -2);
+ mark->mark[k1] = 1;
+ mark->mark[k2] = 1;
+ goto flush;
+ }
- if ( rec1->rec->rid!=rec2->rec->rid ) return 1;
- if ( rec1->rec->pos!=rec2->rec->pos ) return 1;
+ // the last site is not a duplicate with the previous, all sites but the last one can be flushed
+ flush = 1;
+ }
+ else if ( buf->rbuf.n > 1 ) flush = 1;
- return 0;
+flush:
+ if ( !flush ) return 0;
+
+ int i = rbuf_shift(&mark->rbuf);
+ mark->last = mark->mark[i];
+ return 1;
}
-static int _overlap_can_flush(vcfbuf_t *buf, int flush_all)
+static int mark_overlap_helper_(vcfbuf_t *buf, int flush_all)
{
- if ( flush_all ) { buf->overlap.rid = -1; return 1; }
+ if ( buf->status!=dirty ) return flush_all;
- int i = rbuf_last(&buf->rbuf);
- vcfrec_t *last = &buf->vcf[i];
- if ( buf->overlap.rid != last->rec->rid ) buf->overlap.end = 0;
+ int flush = flush_all;
+ mark_t *mark = &buf->mark;
+ // a new site was just added by vcfbuf_push()
+ buf->status = clean;
+
+ rbuf_expand0(&mark->rbuf, uint8_t, buf->rbuf.n, mark->mark);
+ int i = rbuf_append(&mark->rbuf);
+ mark->mark[i] = 0;
+
+ // determine beg and end of the last record that was just added
+ i = rbuf_last(&buf->rbuf);
+ vcfrec_t *last = &buf->vcf[i];
+ if ( mark->overlap_rid != last->rec->rid ) mark->overlap_end = 0;
int beg_pos = last->rec->pos;
int end_pos = last->rec->pos + last->rec->rlen - 1;
// Assuming left-aligned indels. In case it is a deletion, the real variant
- // starts one base after. If an insertion, the overlap with previous zero length.
+ // starts one base after. If an insertion, the overlap with previous is zero
int imin = last->rec->rlen;
for (i=0; i<last->rec->n_allele; i++)
{
while ( *ref && *alt && nt_to_upper(*ref)==nt_to_upper(*alt) ) { ref++; alt++; }
if ( imin > ref - last->rec->d.allele[0] ) imin = ref - last->rec->d.allele[0];
}
-
- if ( beg_pos <= buf->overlap.end )
+ if ( beg_pos <= mark->overlap_end )
{
+ // the new site overlaps with the previous
beg_pos += imin;
if ( beg_pos > end_pos ) end_pos = beg_pos;
}
-
if ( buf->rbuf.n==1 )
{
- buf->overlap.rid = last->rec->rid;
- buf->overlap.end = end_pos;
- return 0;
+ mark->overlap_rid = last->rec->rid;
+ mark->overlap_end = end_pos;
+ return flush;
+ }
+ if ( beg_pos <= mark->overlap_end )
+ {
+ if ( mark->overlap_end < end_pos ) mark->overlap_end = end_pos;
+ int k1 = rbuf_kth(&mark->rbuf, -1);
+ int k2 = rbuf_kth(&mark->rbuf, -2);
+ mark->mark[k1] = 1;
+ mark->mark[k2] = 1;
+ }
+ else
+ {
+ if ( mark->overlap_end < end_pos ) mark->overlap_end = end_pos;
+ flush = 1;
+ }
+ return flush;
+}
+
+
+static int mark_overlap_can_flush_(vcfbuf_t *buf, int flush_all)
+{
+ int flush = flush_all;
+ if ( buf->status==dirty ) flush = mark_overlap_helper_(buf,flush_all);
+ else if ( buf->rbuf.n > 1 ) flush = 1;
+ if ( !flush ) return 0;
+
+ mark_t *mark = &buf->mark;
+ int i = rbuf_shift(&mark->rbuf);
+ mark->last = mark->mark[i];
+ return 1;
+}
+
+
+static int records_overlap(bcf1_t *a, bcf1_t *b)
+{
+ if ( a->rid != b->rid ) return 0;
+ if ( a->pos + a->rlen - 1 < b->pos ) return 0;
+ return 1;
+}
+
+static int cmp_overlap_ptr_asc(const void *aptr, const void *bptr)
+{
+ overlap_t *a = *((overlap_t**)aptr);
+ overlap_t *b = *((overlap_t**)bptr);
+ if ( a->value < b->value ) return -1;
+ if ( a->value > b->value ) return 1;
+ return 0;
+}
+static void mark_expr_missing_reset_(vcfbuf_t *buf)
+{
+ buf->mark.max_qual = 0;
+ buf->mark.max_qual_dp = 0;
+}
+static void mark_expr_missing_prep_(vcfbuf_t *buf, overlap_t *olap)
+{
+ int nval = bcf_get_info_int32(buf->hdr,olap->rec,"DP",&buf->mark.tmpi,&buf->mark.ntmpi);
+ if ( nval!=1 ) return;
+
+ olap->dp = buf->mark.tmpi[0];
+ if ( bcf_float_is_missing(olap->rec->qual) ) return;
+ if ( buf->mark.max_qual < olap->rec->qual )
+ {
+ buf->mark.max_qual = olap->rec->qual;
+ buf->mark.max_qual_dp = olap->dp;
}
- if ( beg_pos <= buf->overlap.end )
+}
+static void mark_expr_missing_set_(vcfbuf_t *buf, overlap_t *olap)
+{
+ if ( !bcf_float_is_missing(olap->rec->qual) ) return;
+ if ( !buf->mark.max_qual_dp ) return;
+
+ // scale QUAL of the most confident variant in the overlap proportionally to the coverage
+ // and use that to prioritize the records
+ olap->value = buf->mark.max_qual * olap->dp / buf->mark.max_qual_dp;
+}
+static int mark_expr_can_flush_(vcfbuf_t *buf, int flush_all)
+{
+ mark_t *mark = &buf->mark;
+ if ( strcasecmp("min(QUAL)",mark->expr) ) error("Todo; at this time only min(QUAL) is supported\n");
+
+ int flush = flush_all;
+ if ( buf->status==dirty )
{
- if ( buf->overlap.end < end_pos ) buf->overlap.end = end_pos;
- return 0;
+ flush = mark_overlap_helper_(buf,flush_all);
+ if ( !flush ) return 0;
+
+ if ( mark->missing_expr==MARK_MISSING_MAX_DP ) mark_expr_missing_reset_(buf);
+
+ // init overlaps, each overlap_t structure keeps a list of overlapping records, symmetrical
+ size_t nori = mark->nbuf;
+ hts_resize(overlap_t, buf->rbuf.n, &mark->nbuf, &mark->buf, HTS_RESIZE_CLEAR);
+ hts_resize(overlap_t*, buf->rbuf.n, &nori, &mark->buf_ptr, HTS_RESIZE_CLEAR);
+ int i;
+ for (i=0; i<buf->rbuf.n; i++)
+ {
+ overlap_t *oi = &mark->buf[i];
+ int j = rbuf_kth(&buf->rbuf, i);
+ assert(j>=0);
+ bcf1_t *rec = buf->vcf[j].rec;
+ assert(rec);
+ oi->rec = rec;
+
+ // todo: other than QUAL values
+ oi->value = bcf_float_is_missing(rec->qual) ? mark->missing_value : rec->qual;
+ if ( mark->missing_expr==MARK_MISSING_MAX_DP ) mark_expr_missing_prep_(buf,oi);
+ if ( oi->bset )
+ {
+ kbs_resize(&oi->bset,buf->rbuf.n);
+ kbs_clear(oi->bset);
+ }
+ else
+ oi->bset = kbs_init(buf->rbuf.n);
+ oi->idx = i;
+ mark->buf_ptr[i] = oi;
+ mark->mark[oi->idx] = 0;
+ }
+ int nolap = 0;
+ for (i=0; i<buf->rbuf.n; i++)
+ {
+ overlap_t *oi = &mark->buf[i];
+ if ( mark->missing_expr==MARK_MISSING_MAX_DP ) mark_expr_missing_set_(buf,oi);
+ int j;
+ for (j=i+1; j<buf->rbuf.n; j++)
+ {
+ overlap_t *oj = &mark->buf[j];
+ if ( !records_overlap(oi->rec,oj->rec) ) continue;
+ kbs_insert(oi->bset,j);
+ kbs_insert(oj->bset,i);
+ nolap++;
+ }
+ }
+
+ // sort according to the requested criteria, currently only min(QUAL)
+ qsort(mark->buf_ptr,buf->rbuf.n,sizeof(*mark->buf_ptr),cmp_overlap_ptr_asc); // todo: other than min()
+
+ // go through the list sorted by overlap_t.value, eg QUAL
+ for (i=0; nolap && i<buf->rbuf.n; i++)
+ {
+ kbitset_iter_t itr;
+ overlap_t *oi = mark->buf_ptr[i];
+ kbs_start(&itr);
+ int j;
+ while ((j=kbs_next(oi->bset, &itr)) >= 0)
+ {
+ kbs_delete(oi->bset,j);
+ assert(nolap);
+ assert(kbs_exists(mark->buf[j].bset,oi->idx));
+ kbs_delete(mark->buf[j].bset,oi->idx);
+ nolap--;
+ }
+ j = rbuf_kth(&mark->rbuf,oi->idx);
+ mark->mark[j] = 1;
+ }
}
+ else if ( buf->rbuf.n > 1 ) flush = 1;
+ if ( !flush ) return 0;
+
+ int i = rbuf_shift(&mark->rbuf);
+ mark->last = mark->mark[i];
return 1;
}
{
int i,j;
+ // nothing to do, no lines in the buffer
if ( buf->rbuf.n==0 ) return NULL;
- if ( flush_all || buf->dummy ) goto ret;
-
- i = rbuf_kth(&buf->rbuf, 0); // first
- j = rbuf_last(&buf->rbuf); // last
- if ( buf->vcf[i].rec->rid != buf->vcf[j].rec->rid ) goto ret;
- if ( buf->overlap.active && _overlap_can_flush(buf, flush_all) ) goto ret;
- if ( buf->rmdup.active && _rmdup_can_flush(buf, flush_all) ) goto ret;
+ // dummy mode, always flushing
+ if ( buf->dummy ) goto ret;
- if ( buf->win > 0 )
+ // pruning mode
+ if ( buf->win )
{
- if ( buf->rbuf.n <= buf->win ) return NULL;
+ int can_flush = flush_all;
+ i = rbuf_kth(&buf->rbuf, 0); // first
+ j = rbuf_last(&buf->rbuf); // last
+ if ( buf->vcf[i].rec->rid != buf->vcf[j].rec->rid ) can_flush = 1;
+ else if ( buf->win > 0 )
+ {
+ if ( buf->rbuf.n > buf->win ) can_flush = 1;
+ }
+ else if ( buf->win < 0 )
+ {
+ if ( !(buf->vcf[i].rec->pos - buf->vcf[j].rec->pos > buf->win) ) can_flush = 1;
+ }
+ buf->status = clean;
+ if ( !can_flush ) return NULL;
+ if ( buf->prune.max_sites && buf->prune.max_sites < buf->rbuf.n ) _prune_sites(buf, flush_all);
goto ret;
}
- else if ( buf->win < 0 )
+
+ // overlaps and duplicates
+ if ( buf->mark.mode )
{
- if ( buf->vcf[i].rec->pos - buf->vcf[j].rec->pos > buf->win ) return NULL;
+ int can_flush = 0;
+ if ( buf->mark.mode==MARK_OVERLAP )
+ {
+ if ( mark_overlap_can_flush_(buf,flush_all) ) can_flush = 1;
+ }
+ else if ( buf->mark.mode==MARK_DUP )
+ {
+ if ( mark_dup_can_flush_(buf,flush_all) ) can_flush = 1;
+ }
+ if ( buf->mark.mode==MARK_EXPR )
+ {
+ if ( mark_expr_can_flush_(buf,flush_all) ) can_flush = 1;
+ }
+ buf->status = clean;
+ if ( !can_flush ) return NULL;
goto ret;
}
- else
- return NULL;
ret:
- if ( buf->prune.max_sites && buf->prune.max_sites < buf->rbuf.n ) _prune_sites(buf, flush_all);
-
+ buf->status = clean;
i = rbuf_shift(&buf->rbuf);
return buf->vcf[i].rec;
}
/* The MIT License
- Copyright (c) 2016-2022 Genome Research Ltd.
+ Copyright (c) 2016-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/vcf.h>
#include <htslib/vcfutils.h>
#include <htslib/hts_os.h>
+#include <htslib/kbitset.h>
#include "bcftools.h"
#include "vcfbuf.h"
#include "rbuf.h"
{
bcf1_t *rec;
double af;
- int af_set:1, filter:1, idx:30;
+ unsigned int af_set:1, filter:1, idx:30;
}
vcfrec_t;
}
prune_t;
+
+#define MARK_OVERLAP 1
+#define MARK_DUP 2
+#define MARK_EXPR 3
+
+#define MARK_MISSING_SCALAR 0 // actual value to use
+#define MARK_MISSING_MAX_DP 1 // max overlap_t.value scaled by INFO/DP
+
+// temporary internal structure for iterative overlap removal by mark_t.expr
typedef struct
{
- int active;
+ double value; // the sort value
+ int rmme, idx; // mark for removal, index in vcfbuf_t.rbuf
+ int dp; // with MARK_MISSING_MAX_DP, INFO/DP is used extrapolate missing QUAL
+ kbitset_t *bset; // mark which records it overlaps with, given as 0-based indexes to vcfbuf_t.rbuf
+ bcf1_t *rec;
}
-rmdup_t;
-
+overlap_t;
typedef struct
{
- int active, rid, end;
+ // modes
+ int mode;
+ char *expr;
+
+ // sites marked according to expr, returned to the caller via vcfbuf_get()
+ rbuf_t rbuf;
+ uint8_t *mark;
+ int last;
+
+ // MARK_OVERLAP
+ int overlap_rid, overlap_end;
+
+ // MARK_EXPR
+ int nbuf;
+ overlap_t *buf, **buf_ptr;
+ int missing_expr; // the value to use when min(QUAL) encounters a missing value
+ float missing_value; // the default missing value
+ float max_qual; // with MARK_MISSING_MAX_DP
+ int max_qual_dp; //
+ int ntmpi; // temporary int array and the allocated memory
+ int32_t *tmpi;
}
-overlap_t;
+mark_t;
struct _vcfbuf_t
{
- int win, dummy;
+ int win, // maximum number of sites in the buffer, either number of sites (<0) or bp (<0)
+ dummy; // the caller maintains the buffer via push/peek/flush
bcf_hdr_t *hdr;
vcfrec_t *vcf;
rbuf_t rbuf;
ld_t ld;
prune_t prune;
- overlap_t overlap;
- rmdup_t rmdup;
+ mark_t mark;
+ enum { clean, dirty } status;
};
vcfbuf_t *vcfbuf_init(bcf_hdr_t *hdr, int win)
vcfbuf_t *buf = (vcfbuf_t*) calloc(1,sizeof(vcfbuf_t));
buf->hdr = hdr;
buf->win = win;
- buf->overlap.rid = -1;
+ buf->status = clean;
+ buf->mark.overlap_rid = -1;
int i;
for (i=0; i<VCFBUF_LD_N; i++) buf->ld.max[i] = HUGE_VAL;
rbuf_init(&buf->rbuf, 0);
free(buf->prune.farr);
free(buf->prune.vrec);
free(buf->prune.ac);
+ free(buf->prune.af_tag);
free(buf->prune.idx);
+ free(buf->mark.mark);
+ free(buf->mark.expr);
+ for (i=0; i<buf->mark.nbuf; i++) kbs_destroy(buf->mark.buf[i].bset);
+ free(buf->mark.buf);
+ free(buf->mark.buf_ptr);
+ free(buf->mark.tmpi);
free(buf);
}
-void vcfbuf_set(vcfbuf_t *buf, vcfbuf_opt_t key, void *value)
+int vcfbuf_set(vcfbuf_t *buf, vcfbuf_opt_t key, ...)
{
- if ( key==LD_FILTER1 ) { buf->ld.filter1 = *((int*)value); return; }
- if ( key==LD_RAND_MISSING ) { buf->ld.rand_missing = *((int*)value); return; }
- if ( key==LD_MAX_R2 ) { buf->ld.max[VCFBUF_LD_IDX_R2] = *((double*)value); return; }
- if ( key==LD_MAX_LD ) { buf->ld.max[VCFBUF_LD_IDX_LD] = *((double*)value); return; }
- if ( key==LD_MAX_HD ) { buf->ld.max[VCFBUF_LD_IDX_HD] = *((double*)value); return; }
-
- if ( key==VCFBUF_DUMMY ) { buf->dummy = *((int*)value); return; }
- if ( key==VCFBUF_NSITES )
+ va_list args;
+ switch (key)
{
- buf->prune.max_sites = *((int*)value);
- if ( !buf->prune.mode ) buf->prune.mode = PRUNE_MODE_MAX_AF;
- return;
+ case LD_FILTER1:
+ va_start(args, key);
+ buf->ld.filter1 = va_arg(args,int);
+ va_end(args);
+ return 0;
+
+ case LD_RAND_MISSING:
+ va_start(args, key);
+ buf->ld.rand_missing = va_arg(args,int);
+ va_end(args);
+ return 0;
+
+ case LD_MAX_R2:
+ va_start(args, key);
+ buf->ld.max[VCFBUF_LD_IDX_R2] = va_arg(args,double);
+ va_end(args);
+ return 0;
+
+ case LD_MAX_LD:
+ va_start(args, key);
+ buf->ld.max[VCFBUF_LD_IDX_LD] = va_arg(args,double);
+ va_end(args);
+ return 0;
+
+ case LD_MAX_HD:
+ va_start(args, key);
+ buf->ld.max[VCFBUF_LD_IDX_HD] = va_arg(args,double);
+ va_end(args);
+ return 0;
+
+ case VCFBUF_DUMMY:
+ va_start(args, key);
+ buf->dummy = va_arg(args,int);
+ va_end(args);
+ return 0;
+
+ case PRUNE_NSITES:
+ va_start(args, key);
+ buf->prune.max_sites = va_arg(args,int);
+ if ( !buf->prune.mode ) buf->prune.mode = PRUNE_MODE_MAX_AF;
+ va_end(args);
+ return 0;
+
+ case PRUNE_NSITES_MODE:
+ va_start(args, key);
+ char *mode = va_arg(args,char*);
+ va_end(args);
+ if ( !strcasecmp(mode,"maxAF") ) buf->prune.mode = PRUNE_MODE_MAX_AF;
+ else if ( !strcasecmp(mode,"1st") ) buf->prune.mode = PRUNE_MODE_1ST;
+ else if ( !strcasecmp(mode,"rand") ) buf->prune.mode = PRUNE_MODE_RAND;
+ else error("The mode \"%s\" is not recognised\n",mode);
+ return 0;
+
+ case PRUNE_AF_TAG:
+ va_start(args, key);
+ buf->prune.af_tag = strdup(va_arg(args,char*));
+ va_end(args);
+ return 0;
+
+ case MARK:
+ va_start(args, key);
+ buf->mark.expr = strdup(va_arg(args,char*));
+ if ( !strcasecmp(buf->mark.expr,"overlap") ) buf->mark.mode = MARK_OVERLAP;
+ else if ( !strcasecmp(buf->mark.expr,"dup") ) buf->mark.mode = MARK_DUP;
+ else buf->mark.mode = MARK_EXPR;
+ va_end(args);
+ return 0;
+
+ case MARK_MISSING_EXPR:
+ va_start(args, key);
+ char *expr = va_arg(args,char*);
+ if ( !strcasecmp(expr,"0") )
+ {
+ buf->mark.missing_expr = MARK_MISSING_SCALAR;
+ buf->mark.missing_value = 0;
+ }
+ else if ( !strcasecmp(expr,"DP") )
+ {
+ if ( buf->mark.mode!=MARK_EXPR ) error("Only the combination of --mark 'min(QUAL)' with --missing DP is currently supported\n");
+ buf->mark.missing_expr = MARK_MISSING_MAX_DP;
+ }
+ else
+ error("todo: MARK_MISSING_EXPR=%s\n",expr);
+ va_end(args);
+ return 0;
}
- if ( key==VCFBUF_AF_TAG ) { buf->prune.af_tag = *((char**)value); return; }
- if ( key==VCFBUF_OVERLAP_WIN ) { buf->overlap.active = *((int*)value); return; }
- if ( key==VCFBUF_RMDUP) { buf->rmdup.active = *((int*)value); return; }
+ return 0;
+}
- if ( key==VCFBUF_NSITES_MODE )
- {
- char *mode = *((char**)value);
- if ( !strcasecmp(mode,"maxAF") ) buf->prune.mode = PRUNE_MODE_MAX_AF;
- else if ( !strcasecmp(mode,"1st") ) buf->prune.mode = PRUNE_MODE_1ST;
- else if ( !strcasecmp(mode,"rand") ) buf->prune.mode = PRUNE_MODE_RAND;
- else error("The mode \"%s\" is not recognised\n",mode);
- return;
- }
+void *vcfbuf_get(vcfbuf_t *buf, vcfbuf_opt_t key, ...)
+{
+ va_list args;
+ va_start(args, key);
+ if ( key==MARK )
+ return &buf->mark.last;
+ va_end(args);
+ return NULL;
}
int vcfbuf_nsites(vcfbuf_t *buf)
bcf1_t *vcfbuf_push(vcfbuf_t *buf, bcf1_t *rec)
{
- rbuf_expand0(&buf->rbuf, vcfrec_t, buf->rbuf.n+1, buf->vcf);
+ // make sure the caller is using the buffer correctly and calls vcfbuf_flush()
+ // before placing next vcfbuf_push() call
+ assert(buf->status!=dirty);
+ if ( !buf->dummy ) buf->status = dirty;
+ rbuf_expand0(&buf->rbuf, vcfrec_t, buf->rbuf.n+1, buf->vcf);
int i = rbuf_append(&buf->rbuf);
if ( !buf->vcf[i].rec ) buf->vcf[i].rec = bcf_init1();
bcf1_t *vcfbuf_peek(vcfbuf_t *buf, int idx)
{
+ buf->status = clean;
int i = rbuf_kth(&buf->rbuf, idx);
return i<0 ? NULL : buf->vcf[i].rec;
}
static void _prune_sites(vcfbuf_t *buf, int flush_all)
{
+
int nbuf = flush_all ? buf->rbuf.n : buf->rbuf.n - 1;
int nprune = nbuf - buf->prune.max_sites;
rbuf_remove_kth(&buf->rbuf, vcfrec_t, buf->prune.idx[i], buf->vcf);
}
-static int _rmdup_can_flush(vcfbuf_t *buf, int flush_all)
+static int mark_dup_can_flush_(vcfbuf_t *buf, int flush_all)
{
- if ( flush_all ) return 1;
+ int flush = flush_all;
+ mark_t *mark = &buf->mark;
+ if ( buf->status==dirty )
+ {
+ // a new site was just added by vcfbuf_push()
+ rbuf_expand0(&mark->rbuf, uint8_t, buf->rbuf.n, mark->mark);
+ int i = rbuf_append(&mark->rbuf);
+ mark->mark[i] = 0;
+
+ if ( buf->rbuf.n==1 ) goto flush;
- if ( buf->rbuf.n==1 ) return 0;
+ // there is at least one previous site, check if it's a duplicate
+ int k1 = rbuf_kth(&buf->rbuf, -1);
+ int k2 = rbuf_kth(&buf->rbuf, -2);
+ vcfrec_t *rec1 = &buf->vcf[k1];
+ vcfrec_t *rec2 = &buf->vcf[k2];
- int k1 = rbuf_kth(&buf->rbuf, -1);
- int k2 = rbuf_kth(&buf->rbuf, -2);
+ int is_dup = 1;
+ if ( rec1->rec->rid!=rec2->rec->rid ) is_dup = 0;
+ else if ( rec1->rec->pos!=rec2->rec->pos ) is_dup = 0;
- vcfrec_t *rec1 = &buf->vcf[k1];
- vcfrec_t *rec2 = &buf->vcf[k2];
+ if ( is_dup )
+ {
+ // it is, mark the last two sites as duplicates
+ int k1 = rbuf_kth(&mark->rbuf, -1);
+ int k2 = rbuf_kth(&mark->rbuf, -2);
+ mark->mark[k1] = 1;
+ mark->mark[k2] = 1;
+ goto flush;
+ }
- if ( rec1->rec->rid!=rec2->rec->rid ) return 1;
- if ( rec1->rec->pos!=rec2->rec->pos ) return 1;
+ // the last site is not a duplicate with the previous, all sites but the last one can be flushed
+ flush = 1;
+ }
+ else if ( buf->rbuf.n > 1 ) flush = 1;
- return 0;
+flush:
+ if ( !flush ) return 0;
+
+ int i = rbuf_shift(&mark->rbuf);
+ mark->last = mark->mark[i];
+ return 1;
}
-static int _overlap_can_flush(vcfbuf_t *buf, int flush_all)
+static int mark_overlap_helper_(vcfbuf_t *buf, int flush_all)
{
- if ( flush_all ) { buf->overlap.rid = -1; return 1; }
+ if ( buf->status!=dirty ) return flush_all;
- int i = rbuf_last(&buf->rbuf);
- vcfrec_t *last = &buf->vcf[i];
- if ( buf->overlap.rid != last->rec->rid ) buf->overlap.end = 0;
+ int flush = flush_all;
+ mark_t *mark = &buf->mark;
+ // a new site was just added by vcfbuf_push()
+ buf->status = clean;
+
+ rbuf_expand0(&mark->rbuf, uint8_t, buf->rbuf.n, mark->mark);
+ int i = rbuf_append(&mark->rbuf);
+ mark->mark[i] = 0;
+
+ // determine beg and end of the last record that was just added
+ i = rbuf_last(&buf->rbuf);
+ vcfrec_t *last = &buf->vcf[i];
+ if ( mark->overlap_rid != last->rec->rid ) mark->overlap_end = 0;
int beg_pos = last->rec->pos;
int end_pos = last->rec->pos + last->rec->rlen - 1;
// Assuming left-aligned indels. In case it is a deletion, the real variant
- // starts one base after. If an insertion, the overlap with previous zero length.
+ // starts one base after. If an insertion, the overlap with previous is zero
int imin = last->rec->rlen;
for (i=0; i<last->rec->n_allele; i++)
{
while ( *ref && *alt && nt_to_upper(*ref)==nt_to_upper(*alt) ) { ref++; alt++; }
if ( imin > ref - last->rec->d.allele[0] ) imin = ref - last->rec->d.allele[0];
}
-
- if ( beg_pos <= buf->overlap.end )
+ if ( beg_pos <= mark->overlap_end )
{
+ // the new site overlaps with the previous
beg_pos += imin;
if ( beg_pos > end_pos ) end_pos = beg_pos;
}
-
if ( buf->rbuf.n==1 )
{
- buf->overlap.rid = last->rec->rid;
- buf->overlap.end = end_pos;
- return 0;
+ mark->overlap_rid = last->rec->rid;
+ mark->overlap_end = end_pos;
+ return flush;
+ }
+ if ( beg_pos <= mark->overlap_end )
+ {
+ if ( mark->overlap_end < end_pos ) mark->overlap_end = end_pos;
+ int k1 = rbuf_kth(&mark->rbuf, -1);
+ int k2 = rbuf_kth(&mark->rbuf, -2);
+ mark->mark[k1] = 1;
+ mark->mark[k2] = 1;
+ }
+ else
+ {
+ if ( mark->overlap_end < end_pos ) mark->overlap_end = end_pos;
+ flush = 1;
+ }
+ return flush;
+}
+
+
+static int mark_overlap_can_flush_(vcfbuf_t *buf, int flush_all)
+{
+ int flush = flush_all;
+ if ( buf->status==dirty ) flush = mark_overlap_helper_(buf,flush_all);
+ else if ( buf->rbuf.n > 1 ) flush = 1;
+ if ( !flush ) return 0;
+
+ mark_t *mark = &buf->mark;
+ int i = rbuf_shift(&mark->rbuf);
+ mark->last = mark->mark[i];
+ return 1;
+}
+
+
+static int records_overlap(bcf1_t *a, bcf1_t *b)
+{
+ if ( a->rid != b->rid ) return 0;
+ if ( a->pos + a->rlen - 1 < b->pos ) return 0;
+ return 1;
+}
+
+static int cmp_overlap_ptr_asc(const void *aptr, const void *bptr)
+{
+ overlap_t *a = *((overlap_t**)aptr);
+ overlap_t *b = *((overlap_t**)bptr);
+ if ( a->value < b->value ) return -1;
+ if ( a->value > b->value ) return 1;
+ return 0;
+}
+static void mark_expr_missing_reset_(vcfbuf_t *buf)
+{
+ buf->mark.max_qual = 0;
+ buf->mark.max_qual_dp = 0;
+}
+static void mark_expr_missing_prep_(vcfbuf_t *buf, overlap_t *olap)
+{
+ int nval = bcf_get_info_int32(buf->hdr,olap->rec,"DP",&buf->mark.tmpi,&buf->mark.ntmpi);
+ if ( nval!=1 ) return;
+
+ olap->dp = buf->mark.tmpi[0];
+ if ( bcf_float_is_missing(olap->rec->qual) ) return;
+ if ( buf->mark.max_qual < olap->rec->qual )
+ {
+ buf->mark.max_qual = olap->rec->qual;
+ buf->mark.max_qual_dp = olap->dp;
}
- if ( beg_pos <= buf->overlap.end )
+}
+static void mark_expr_missing_set_(vcfbuf_t *buf, overlap_t *olap)
+{
+ if ( !bcf_float_is_missing(olap->rec->qual) ) return;
+ if ( !buf->mark.max_qual_dp ) return;
+
+ // scale QUAL of the most confident variant in the overlap proportionally to the coverage
+ // and use that to prioritize the records
+ olap->value = buf->mark.max_qual * olap->dp / buf->mark.max_qual_dp;
+}
+static int mark_expr_can_flush_(vcfbuf_t *buf, int flush_all)
+{
+ mark_t *mark = &buf->mark;
+ if ( strcasecmp("min(QUAL)",mark->expr) ) error("Todo; at this time only min(QUAL) is supported\n");
+
+ int flush = flush_all;
+ if ( buf->status==dirty )
{
- if ( buf->overlap.end < end_pos ) buf->overlap.end = end_pos;
- return 0;
+ flush = mark_overlap_helper_(buf,flush_all);
+ if ( !flush ) return 0;
+
+ if ( mark->missing_expr==MARK_MISSING_MAX_DP ) mark_expr_missing_reset_(buf);
+
+ // init overlaps, each overlap_t structure keeps a list of overlapping records, symmetrical
+ size_t nori = mark->nbuf;
+ hts_resize(overlap_t, buf->rbuf.n, &mark->nbuf, &mark->buf, HTS_RESIZE_CLEAR);
+ hts_resize(overlap_t*, buf->rbuf.n, &nori, &mark->buf_ptr, HTS_RESIZE_CLEAR);
+ int i;
+ for (i=0; i<buf->rbuf.n; i++)
+ {
+ overlap_t *oi = &mark->buf[i];
+ int j = rbuf_kth(&buf->rbuf, i);
+ assert(j>=0);
+ bcf1_t *rec = buf->vcf[j].rec;
+ assert(rec);
+ oi->rec = rec;
+
+ // todo: other than QUAL values
+ oi->value = bcf_float_is_missing(rec->qual) ? mark->missing_value : rec->qual;
+ if ( mark->missing_expr==MARK_MISSING_MAX_DP ) mark_expr_missing_prep_(buf,oi);
+ if ( oi->bset )
+ {
+ kbs_resize(&oi->bset,buf->rbuf.n);
+ kbs_clear(oi->bset);
+ }
+ else
+ oi->bset = kbs_init(buf->rbuf.n);
+ oi->idx = i;
+ mark->buf_ptr[i] = oi;
+ mark->mark[oi->idx] = 0;
+ }
+ int nolap = 0;
+ for (i=0; i<buf->rbuf.n; i++)
+ {
+ overlap_t *oi = &mark->buf[i];
+ if ( mark->missing_expr==MARK_MISSING_MAX_DP ) mark_expr_missing_set_(buf,oi);
+ int j;
+ for (j=i+1; j<buf->rbuf.n; j++)
+ {
+ overlap_t *oj = &mark->buf[j];
+ if ( !records_overlap(oi->rec,oj->rec) ) continue;
+ kbs_insert(oi->bset,j);
+ kbs_insert(oj->bset,i);
+ nolap++;
+ }
+ }
+
+ // sort according to the requested criteria, currently only min(QUAL)
+ qsort(mark->buf_ptr,buf->rbuf.n,sizeof(*mark->buf_ptr),cmp_overlap_ptr_asc); // todo: other than min()
+
+ // go through the list sorted by overlap_t.value, eg QUAL
+ for (i=0; nolap && i<buf->rbuf.n; i++)
+ {
+ kbitset_iter_t itr;
+ overlap_t *oi = mark->buf_ptr[i];
+ kbs_start(&itr);
+ int j;
+ while ((j=kbs_next(oi->bset, &itr)) >= 0)
+ {
+ kbs_delete(oi->bset,j);
+ assert(nolap);
+ assert(kbs_exists(mark->buf[j].bset,oi->idx));
+ kbs_delete(mark->buf[j].bset,oi->idx);
+ nolap--;
+ }
+ j = rbuf_kth(&mark->rbuf,oi->idx);
+ mark->mark[j] = 1;
+ }
}
+ else if ( buf->rbuf.n > 1 ) flush = 1;
+ if ( !flush ) return 0;
+
+ int i = rbuf_shift(&mark->rbuf);
+ mark->last = mark->mark[i];
return 1;
}
{
int i,j;
+ // nothing to do, no lines in the buffer
if ( buf->rbuf.n==0 ) return NULL;
- if ( flush_all || buf->dummy ) goto ret;
-
- i = rbuf_kth(&buf->rbuf, 0); // first
- j = rbuf_last(&buf->rbuf); // last
- if ( buf->vcf[i].rec->rid != buf->vcf[j].rec->rid ) goto ret;
- if ( buf->overlap.active && _overlap_can_flush(buf, flush_all) ) goto ret;
- if ( buf->rmdup.active && _rmdup_can_flush(buf, flush_all) ) goto ret;
+ // dummy mode, always flushing
+ if ( buf->dummy ) goto ret;
- if ( buf->win > 0 )
+ // pruning mode
+ if ( buf->win )
{
- if ( buf->rbuf.n <= buf->win ) return NULL;
+ int can_flush = flush_all;
+ i = rbuf_kth(&buf->rbuf, 0); // first
+ j = rbuf_last(&buf->rbuf); // last
+ if ( buf->vcf[i].rec->rid != buf->vcf[j].rec->rid ) can_flush = 1;
+ else if ( buf->win > 0 )
+ {
+ if ( buf->rbuf.n > buf->win ) can_flush = 1;
+ }
+ else if ( buf->win < 0 )
+ {
+ if ( !(buf->vcf[i].rec->pos - buf->vcf[j].rec->pos > buf->win) ) can_flush = 1;
+ }
+ buf->status = clean;
+ if ( !can_flush ) return NULL;
+ if ( buf->prune.max_sites && buf->prune.max_sites < buf->rbuf.n ) _prune_sites(buf, flush_all);
goto ret;
}
- else if ( buf->win < 0 )
+
+ // overlaps and duplicates
+ if ( buf->mark.mode )
{
- if ( buf->vcf[i].rec->pos - buf->vcf[j].rec->pos > buf->win ) return NULL;
+ int can_flush = 0;
+ if ( buf->mark.mode==MARK_OVERLAP )
+ {
+ if ( mark_overlap_can_flush_(buf,flush_all) ) can_flush = 1;
+ }
+ else if ( buf->mark.mode==MARK_DUP )
+ {
+ if ( mark_dup_can_flush_(buf,flush_all) ) can_flush = 1;
+ }
+ if ( buf->mark.mode==MARK_EXPR )
+ {
+ if ( mark_expr_can_flush_(buf,flush_all) ) can_flush = 1;
+ }
+ buf->status = clean;
+ if ( !can_flush ) return NULL;
goto ret;
}
- else
- return NULL;
ret:
- if ( buf->prune.max_sites && buf->prune.max_sites < buf->rbuf.n ) _prune_sites(buf, flush_all);
-
+ buf->status = clean;
i = rbuf_shift(&buf->rbuf);
return buf->vcf[i].rec;
}
/* The MIT License
- Copyright (c) 2017-2022 Genome Research Ltd.
+ Copyright (c) 2017-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
// Modes of operation
typedef enum
{
- VCFBUF_DUMMY, // the caller maintains the buffer via push/peek/flush, nothing is removed by vcfbuf
-
- VCFBUF_OVERLAP_WIN, // keep only overlapping variants in the window
- VCFBUF_RMDUP, // remove duplicate sites (completely)
- VCFBUF_NSITES, // leave at max this many sites in the window
- VCFBUF_NSITES_MODE, // one of: maxAF (keep sites with max AF), 1st (sites that come first), rand (pick randomly)
- VCFBUF_AF_TAG, // use this INFO tag with VCFBUF_NSITES
+ VCFBUF_DUMMY, // int {0,1}, the caller maintains the buffer via push/peek/flush, nothing is removed by vcfbuf
+
+ // pruning
+ PRUNE_NSITES, // int, leave max this many sites in the window
+ PRUNE_NSITES_MODE, // char *, maxAF (keep sites with max AF), 1st (sites that come first), rand (pick randomly)
+ PRUNE_AF_TAG, // char *, use this INFO/AF tag with VCFBUF_NSITES
+
+ // duplicates and overlaps
+ MARK, // w: char *, resolve overlaps by preferentially removing sites according to EXPR:
+ // min(QUAL) .. remove sites with lowest QUAL until overlaps are resolved
+ // overlap .. select all overlapping sites
+ // dup .. select duplicate sites
+ // r: use as
+ // while ( (rec=vcfbuf_flush(buf,flush_all)) )
+ // {
+ // int is_marked = vcfbuf_get_val(buf,int,MARK);
+ // if ( is_marked ) do_something(rec);
+ // }
+ MARK_MISSING_EXPR, // char *, what to do when missing value are encountered with min(QUAL)
+ // 0 .. set to 0 (the default)
+ // DP .. scale max quality in the window proportionally to INFO/DP
// LD related options
LD_RAND_MISSING, // randomize rather than ignore missing genotypes
}
vcfbuf_opt_t;
-#define vcfbuf_set_opt(buf,type,key,value) { type tmp = value; vcfbuf_set(buf, key, (void*)&tmp); }
-void vcfbuf_set(vcfbuf_t *buf, vcfbuf_opt_t key, void *value);
+
+/**
+ * vcfbuf_set() - set various options, see the vcfbuf_opt_t keys for the complete list
+ *
+ * Returns 0 if the call succeeded, or negative number on error.
+ */
+int vcfbuf_set(vcfbuf_t *buf, vcfbuf_opt_t key, ...); // returns 0 on success
+
+/**
+ * vcfbuf_get() - get various options, see the vcfbuf_opt_t keys
+ * vcfbuf_get_val() - wrapper for `vcfbuf_get()` to return typed value
+ *
+ * The former returns pointer to the memory area populated by the requested setting,
+ * its type can be inferred from the vcfbuf_opt_t documentation.
+ */
+void *vcfbuf_get(vcfbuf_t *buf, vcfbuf_opt_t key, ...);
+#define vcfbuf_get_val(buf,type,key) (*(type*)vcfbuf_get(buf, key))
/*
void vcfbuf_destroy(vcfbuf_t *buf);
/*
- * vcfbuf_push() - push a new site for analysis
+ * vcfbuf_push() - push a new site for analysis.
+ *
+ * Note that vcfbuf_flush() or vcfbuf_peek() must be called before next site is pushed.
*/
bcf1_t *vcfbuf_push(vcfbuf_t *buf, bcf1_t *rec);
/*
* vcfbuf_flush() - returns the next record or NULL, depending on the mode of operation and
* the content of the buffer
+ *
+ * @flush_all: 1 if no more vcfbuf_push() calls will follow, 0 otherwise
*/
bcf1_t *vcfbuf_flush(vcfbuf_t *buf, int flush_all);
/* vcfcall.c -- SNP/indel variant calling from VCF/BCF.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
while ( *ss )
{
se = ss;
- while ( *se && *se!=',' ) se++;
+ while ( *se && *se!=',' && !isspace(*se) ) se++;
als->n++;
als->allele = (char**)realloc(als->allele,als->n*sizeof(*als->allele));
als->allele[als->n-1] = (char*)malloc(se-ss+1);
memcpy(als->allele[als->n-1],ss,se-ss);
als->allele[als->n-1][se-ss] = 0;
ss = se+1;
- if ( !*se ) break;
+ if ( !*se || isspace(*se) ) break;
}
+ if ( als->n<2 ) error("Unable to parse the -T file; expected CHROM\\tPOS\\tREF,ALT with -C alleles but found instead:\n\t%s\n",line);
return 0;
}
static void tgt_free(void *payload)
}
if ( args->aux.flag & CALL_CONSTR_ALLELES )
+ {
args->vcfbuf = vcfbuf_init(args->aux.hdr, 0);
+ vcfbuf_set(args->vcfbuf,VCFBUF_DUMMY,1);
+ }
char wmode[8];
set_wmode(wmode,args->output_type,args->output_fname,args->clevel);
if (args->record_cmd_line) bcf_hdr_append_version(args->aux.hdr, args->argc, args->argv, "bcftools_call");
if ( bcf_hdr_write(args->out_fh, args->aux.hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->aux.hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->aux.hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
if ( args->flag&CF_INS_MISSED ) init_missed_line(args);
}
fprintf(stderr, "\n");
fprintf(stderr, "Input/output options:\n");
fprintf(stderr, " -A, --keep-alts Keep all possible alternate alleles at variant sites\n");
+ fprintf(stderr, " -*, --keep-unseen-allele Keep the unobserved allele <*> or <NON_REF>\n");
fprintf(stderr, " -a, --annotate LIST Optional tags to output (lowercase allowed); '?' to list available tags\n");
fprintf(stderr, " -F, --prior-freqs AN,AC Use prior allele frequencies, determined from these pre-filled tags\n");
fprintf(stderr, " -G, --group-samples FILE|- Group samples by population (file with \"sample\\tgroup\") or \"-\" for single-sample calling.\n");
fprintf(stderr, " -M, --keep-masked-ref Keep sites with masked reference allele (REF=N)\n");
fprintf(stderr, " -V, --skip-variants TYPE Skip indels/snps\n");
fprintf(stderr, " -v, --variants-only Output variant sites only\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Consensus/variant calling options:\n");
fprintf(stderr, " -c, --consensus-caller The original calling method (conflicts with -m)\n");
// todo (and more)
// fprintf(stderr, "\nContrast calling and association test options:\n");
// fprintf(stderr, " -1 INT number of group-1 samples [0]\n");
- // fprintf(stderr, " -C FLOAT posterior constrast for LRT<FLOAT and P(ref|D)<0.5 [%g]\n", args->aux.min_lrt);
+ // fprintf(stderr, " -C FLOAT posterior contrast for LRT<FLOAT and P(ref|D)<0.5 [%g]\n", args->aux.min_lrt);
// fprintf(stderr, " -U INT number of permutations for association testing (effective with -1) [0]\n");
// fprintf(stderr, " -X FLOAT only perform permutations for P(chi^2)<FLOAT [%g]\n", args->aux.min_perm_p);
fprintf(stderr, "\n");
{"targets-file",required_argument,NULL,'T'},
{"threads",required_argument,NULL,9},
{"keep-alts",no_argument,NULL,'A'},
+ {"keep-unseen-allele",no_argument,NULL,'*'},
{"insert-missed",no_argument,NULL,'i'},
{"skip-Ns",no_argument,NULL,'N'}, // now the new default
{"keep-masked-refs",no_argument,NULL,'M'},
{"chromosome-X",no_argument,NULL,'X'},
{"chromosome-Y",no_argument,NULL,'Y'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp = NULL;
- while ((c = getopt_long(argc, argv, "h?o:O:r:R:s:S:t:T:ANMV:vcmp:C:n:P:f:a:ig:XYF:G:", loptions, NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h?o:O:r:R:s:S:t:T:A*NMV:vcmp:C:n:P:f:a:ig:XYF:G:W::", loptions, NULL)) >= 0)
{
switch (c)
{
case 'M': args.flag &= ~CF_ACGT_ONLY; break; // keep sites where REF is N
case 'N': args.flag |= CF_ACGT_ONLY; break; // omit sites where first base in REF is N (the new default)
case 'A': args.aux.flag |= CALL_KEEPALT; break;
+ case '*': args.aux.flag |= CALL_KEEP_UNSEEN; break;
case 'c': args.flag |= CF_CCALL; break; // the original EM based calling method
case 'i': args.flag |= CF_INS_MISSED; break;
case 'v': args.aux.flag |= CALL_VARONLY; break;
args.regions_overlap = parse_overlap_option(optarg);
if ( args.regions_overlap < 0 ) error("Could not parse: --regions-overlap %s\n",optarg);
break;
- case 10: args.write_index = 1; break;
+ case 'W':
+ if (!(args.write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
default: usage(&args);
}
}
/* vcfcall.c -- SNP/indel variant calling from VCF/BCF.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
while ( *ss )
{
se = ss;
- while ( *se && *se!=',' ) se++;
+ while ( *se && *se!=',' && !isspace(*se) ) se++;
als->n++;
als->allele = (char**)realloc(als->allele,als->n*sizeof(*als->allele));
als->allele[als->n-1] = (char*)malloc(se-ss+1);
memcpy(als->allele[als->n-1],ss,se-ss);
als->allele[als->n-1][se-ss] = 0;
ss = se+1;
- if ( !*se ) break;
+ if ( !*se || isspace(*se) ) break;
}
+ if ( als->n<2 ) error("Unable to parse the -T file; expected CHROM\\tPOS\\tREF,ALT with -C alleles but found instead:\n\t%s\n",line);
return 0;
}
static void tgt_free(void *payload)
}
if ( args->aux.flag & CALL_CONSTR_ALLELES )
+ {
args->vcfbuf = vcfbuf_init(args->aux.hdr, 0);
+ vcfbuf_set(args->vcfbuf,VCFBUF_DUMMY,1);
+ }
char wmode[8];
set_wmode(wmode,args->output_type,args->output_fname,args->clevel);
if (args->record_cmd_line) bcf_hdr_append_version(args->aux.hdr, args->argc, args->argv, "bcftools_call");
if ( bcf_hdr_write(args->out_fh, args->aux.hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->aux.hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->aux.hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
if ( args->flag&CF_INS_MISSED ) init_missed_line(args);
}
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Input/output options:\n");
fprintf(bcftools_stderr, " -A, --keep-alts Keep all possible alternate alleles at variant sites\n");
+ fprintf(bcftools_stderr, " -*, --keep-unseen-allele Keep the unobserved allele <*> or <NON_REF>\n");
fprintf(bcftools_stderr, " -a, --annotate LIST Optional tags to output (lowercase allowed); '?' to list available tags\n");
fprintf(bcftools_stderr, " -F, --prior-freqs AN,AC Use prior allele frequencies, determined from these pre-filled tags\n");
fprintf(bcftools_stderr, " -G, --group-samples FILE|- Group samples by population (file with \"sample\\tgroup\") or \"-\" for single-sample calling.\n");
fprintf(bcftools_stderr, " -M, --keep-masked-ref Keep sites with masked reference allele (REF=N)\n");
fprintf(bcftools_stderr, " -V, --skip-variants TYPE Skip indels/snps\n");
fprintf(bcftools_stderr, " -v, --variants-only Output variant sites only\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Consensus/variant calling options:\n");
fprintf(bcftools_stderr, " -c, --consensus-caller The original calling method (conflicts with -m)\n");
// todo (and more)
// fprintf(bcftools_stderr, "\nContrast calling and association test options:\n");
// fprintf(bcftools_stderr, " -1 INT number of group-1 samples [0]\n");
- // fprintf(bcftools_stderr, " -C FLOAT posterior constrast for LRT<FLOAT and P(ref|D)<0.5 [%g]\n", args->aux.min_lrt);
+ // fprintf(bcftools_stderr, " -C FLOAT posterior contrast for LRT<FLOAT and P(ref|D)<0.5 [%g]\n", args->aux.min_lrt);
// fprintf(bcftools_stderr, " -U INT number of permutations for association testing (effective with -1) [0]\n");
// fprintf(bcftools_stderr, " -X FLOAT only perform permutations for P(chi^2)<FLOAT [%g]\n", args->aux.min_perm_p);
fprintf(bcftools_stderr, "\n");
{"targets-file",required_argument,NULL,'T'},
{"threads",required_argument,NULL,9},
{"keep-alts",no_argument,NULL,'A'},
+ {"keep-unseen-allele",no_argument,NULL,'*'},
{"insert-missed",no_argument,NULL,'i'},
{"skip-Ns",no_argument,NULL,'N'}, // now the new default
{"keep-masked-refs",no_argument,NULL,'M'},
{"chromosome-X",no_argument,NULL,'X'},
{"chromosome-Y",no_argument,NULL,'Y'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp = NULL;
- while ((c = getopt_long(argc, argv, "h?o:O:r:R:s:S:t:T:ANMV:vcmp:C:n:P:f:a:ig:XYF:G:", loptions, NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h?o:O:r:R:s:S:t:T:A*NMV:vcmp:C:n:P:f:a:ig:XYF:G:W::", loptions, NULL)) >= 0)
{
switch (c)
{
case 'M': args.flag &= ~CF_ACGT_ONLY; break; // keep sites where REF is N
case 'N': args.flag |= CF_ACGT_ONLY; break; // omit sites where first base in REF is N (the new default)
case 'A': args.aux.flag |= CALL_KEEPALT; break;
+ case '*': args.aux.flag |= CALL_KEEP_UNSEEN; break;
case 'c': args.flag |= CF_CCALL; break; // the original EM based calling method
case 'i': args.flag |= CF_INS_MISSED; break;
case 'v': args.aux.flag |= CALL_VARONLY; break;
args.regions_overlap = parse_overlap_option(optarg);
if ( args.regions_overlap < 0 ) error("Could not parse: --regions-overlap %s\n",optarg);
break;
- case 10: args.write_index = 1; break;
+ case 'W':
+ if (!(args.write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
default: usage(&args);
}
}
#include <htslib/kstring.h>
#include <htslib/kfunc.h>
#include <htslib/khash_str2int.h>
+#include <htslib/hts_defs.h>
#include "bcftools.h"
#include "HMM.h"
#include "rbuf.h"
}
args_t;
-FILE *open_file(char **fname, const char *mode, const char *fmt, ...);
+FILE *open_file(char **fname, const char *mode, const char *fmt, ...) HTS_FORMAT(HTS_PRINTF_FMT, 3, 4);
static inline void hmm2cn_state(int nstates, int i, int *a, int *b)
{
/*
A noisy CN2 band is hard to distinguish from two CN3 bands which are
- close to each other. Set a treshold on the minimum separation based
+ close to each other. Set a threshold on the minimum separation based
on the BAF deviation at p=0.95
*/
baf_dev2 /= norm_cn3;
{0,0,0,0}
};
char *tmp = NULL;
- while ((c = getopt_long(argc, argv, "h?r:R:t:T:s:o:p:l:T:c:b:P:x:e:O:W:f:a:L:d:k:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "h?r:R:t:T:s:o:p:l:T:c:b:P:x:e:O:W::f:a:L:d:k:",loptions,NULL)) >= 0) {
switch (c) {
case 'L':
args->lrr_smooth_win = strtol(optarg,&tmp,10);
#include <htslib/kstring.h>
#include <htslib/kfunc.h>
#include <htslib/khash_str2int.h>
+#include <htslib/hts_defs.h>
#include "bcftools.h"
#include "HMM.h"
#include "rbuf.h"
}
args_t;
-FILE *open_file(char **fname, const char *mode, const char *fmt, ...);
+FILE *open_file(char **fname, const char *mode, const char *fmt, ...) HTS_FORMAT(HTS_PRINTF_FMT, 3, 4);
static inline void hmm2cn_state(int nstates, int i, int *a, int *b)
{
/*
A noisy CN2 band is hard to distinguish from two CN3 bands which are
- close to each other. Set a treshold on the minimum separation based
+ close to each other. Set a threshold on the minimum separation based
on the BAF deviation at p=0.95
*/
baf_dev2 /= norm_cn3;
{0,0,0,0}
};
char *tmp = NULL;
- while ((c = getopt_long(argc, argv, "h?r:R:t:T:s:o:p:l:T:c:b:P:x:e:O:W:f:a:L:d:k:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "h?r:R:t:T:s:o:p:l:T:c:b:P:x:e:O:W::f:a:L:d:k:",loptions,NULL)) >= 0) {
switch (c) {
case 'L':
args->lrr_smooth_win = strtol(optarg,&tmp,10);
#include <assert.h>
#include <errno.h>
#include <math.h>
+#include <stdint.h>
#include <inttypes.h>
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
#include <htslib/bgzf.h>
#include <htslib/tbx.h> // for hts_get_bgzfp()
#include <htslib/thread_pool.h>
+#include <htslib/hts_endian.h>
#include <sys/time.h>
#include "bcftools.h"
hts_set_opt(args->out_fh, HTS_OPT_THREAD_POOL, args->tpool);
}
if ( bcf_hdr_write(args->out_fh, args->out_hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->out_hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->out_hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
if ( args->allow_overlaps )
{
for (j=0; j<hdr0->nhrec; j++)
{
bcf_hrec_t *hrec0 = hdr0->hrec[j];
- if ( hrec0->type!=BCF_HL_FLT && hrec0->type!=BCF_HL_INFO && hrec0->type!=BCF_HL_FMT && hrec0->type!=BCF_HL_CTG ) continue; // skip fiels w/o IDX
+ if ( hrec0->type!=BCF_HL_FLT && hrec0->type!=BCF_HL_INFO && hrec0->type!=BCF_HL_FMT && hrec0->type!=BCF_HL_CTG ) continue; // skip fields w/o IDX
int itag = bcf_hrec_find_key(hrec0, "ID");
bcf_hrec_t *hrec = bcf_hdr_get_hrec(hdr, hrec0->type, "ID", hrec0->vals[itag], NULL);
int nskip;
if ( type.format==bcf )
{
- uint8_t magic[5];
- if ( bgzf_read(fp, magic, 5) != 5 ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
+ const size_t magic_len = 5 + 4; // "Magic" string + header length
+ uint8_t magic[magic_len];
+ if ( bgzf_read(fp, magic, magic_len) != magic_len ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
+ // First five bytes are the "Magic" string
if (strncmp((char*)magic, "BCF\2\2", 5) != 0) error("\nInvalid BCF magic string in %s\n", args->fnames[i]);
-
- if ( bgzf_read(fp, &tmp.l, 4) != 4 ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
+ // Next four are the header length (little-endian)
+ tmp.l = le_to_u32(magic + 5);
hts_expand(char,tmp.l,tmp.m,tmp.s);
if ( bgzf_read(fp, tmp.s, tmp.l) != tmp.l ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
// write only the first header
if ( i==0 )
{
- if ( bgzf_write(bgzf_out, "BCF\2\2", 5) !=5 ) error("\nFailed to write %d bytes to %s\n", 5,args->output_fname);
- if ( bgzf_write(bgzf_out, &tmp.l, 4) !=4 ) error("\nFailed to write %d bytes to %s\n", 4,args->output_fname);
+ if ( bgzf_write(bgzf_out, magic, magic_len) != magic_len ) error("\nFailed to write %zu bytes to %s\n", magic_len,args->output_fname);
if ( bgzf_write(bgzf_out, tmp.s, tmp.l) != tmp.l) error("\nFailed to write %"PRId64" bytes to %s\n", (uint64_t)tmp.l,args->output_fname);
}
nskip = fp->block_offset;
fprintf(stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
fprintf(stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
fprintf(stderr, " -v, --verbose 0|1 Set verbosity level [1]\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
exit(1);
}
{"file-list",required_argument,NULL,'f'},
{"min-PQ",required_argument,NULL,'q'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,13},
+ {"write-index",optional_argument,NULL,'W'},
{"drop-genotypes",no_argument,NULL,'G'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "h:?o:O:f:alq:Dd:Gr:R:cnv:",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h:?o:O:f:alq:Dd:Gr:R:cnv:W::",loptions,NULL)) >= 0)
{
switch (c) {
case 'c': args->compact_PS = 1; break;
args->verbose = strtol(optarg, &tmp, 0);
if ( *tmp || args->verbose<0 || args->verbose>1 ) error("Error: currently only --verbose 0 or --verbose 1 is supported\n");
break;
- case 13 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
if ( args->regions_list && !args->allow_overlaps ) error("The -r/-R option is supported only with -a\n");
if ( args->naive_concat )
{
+ if ( args->write_index ) error("Error: cannot --write-index in the %s mode\n",args->naive_concat_trust_headers?"--naive-force":"--naive");
if ( args->allow_overlaps ) error("The option --naive cannot be combined with --allow-overlaps\n");
if ( args->phased_concat ) error("The option --naive cannot be combined with --ligate\n");
if ( args->sites_only ) error("The option --naive cannot be combined with --drop-genotypes\n");
#include <assert.h>
#include <errno.h>
#include <math.h>
+#include <stdint.h>
#include <inttypes.h>
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
#include <htslib/bgzf.h>
#include <htslib/tbx.h> // for hts_get_bgzfp()
#include <htslib/thread_pool.h>
+#include <htslib/hts_endian.h>
#include <sys/time.h>
#include "bcftools.h"
hts_set_opt(args->out_fh, HTS_OPT_THREAD_POOL, args->tpool);
}
if ( bcf_hdr_write(args->out_fh, args->out_hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->out_hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->out_hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
if ( args->allow_overlaps )
{
for (j=0; j<hdr0->nhrec; j++)
{
bcf_hrec_t *hrec0 = hdr0->hrec[j];
- if ( hrec0->type!=BCF_HL_FLT && hrec0->type!=BCF_HL_INFO && hrec0->type!=BCF_HL_FMT && hrec0->type!=BCF_HL_CTG ) continue; // skip fiels w/o IDX
+ if ( hrec0->type!=BCF_HL_FLT && hrec0->type!=BCF_HL_INFO && hrec0->type!=BCF_HL_FMT && hrec0->type!=BCF_HL_CTG ) continue; // skip fields w/o IDX
int itag = bcf_hrec_find_key(hrec0, "ID");
bcf_hrec_t *hrec = bcf_hdr_get_hrec(hdr, hrec0->type, "ID", hrec0->vals[itag], NULL);
int nskip;
if ( type.format==bcf )
{
- uint8_t magic[5];
- if ( bgzf_read(fp, magic, 5) != 5 ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
+ const size_t magic_len = 5 + 4; // "Magic" string + header length
+ uint8_t magic[magic_len];
+ if ( bgzf_read(fp, magic, magic_len) != magic_len ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
+ // First five bytes are the "Magic" string
if (strncmp((char*)magic, "BCF\2\2", 5) != 0) error("\nInvalid BCF magic string in %s\n", args->fnames[i]);
-
- if ( bgzf_read(fp, &tmp.l, 4) != 4 ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
+ // Next four are the header length (little-endian)
+ tmp.l = le_to_u32(magic + 5);
hts_expand(char,tmp.l,tmp.m,tmp.s);
if ( bgzf_read(fp, tmp.s, tmp.l) != tmp.l ) error("\nFailed to read the BCF header in %s\n", args->fnames[i]);
// write only the first header
if ( i==0 )
{
- if ( bgzf_write(bgzf_out, "BCF\2\2", 5) !=5 ) error("\nFailed to write %d bytes to %s\n", 5,args->output_fname);
- if ( bgzf_write(bgzf_out, &tmp.l, 4) !=4 ) error("\nFailed to write %d bytes to %s\n", 4,args->output_fname);
+ if ( bgzf_write(bgzf_out, magic, magic_len) != magic_len ) error("\nFailed to write %zu bytes to %s\n", magic_len,args->output_fname);
if ( bgzf_write(bgzf_out, tmp.s, tmp.l) != tmp.l) error("\nFailed to write %"PRId64" bytes to %s\n", (uint64_t)tmp.l,args->output_fname);
}
nskip = fp->block_offset;
fprintf(bcftools_stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
fprintf(bcftools_stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
fprintf(bcftools_stderr, " -v, --verbose 0|1 Set verbosity level [1]\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
{"file-list",required_argument,NULL,'f'},
{"min-PQ",required_argument,NULL,'q'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,13},
+ {"write-index",optional_argument,NULL,'W'},
{"drop-genotypes",no_argument,NULL,'G'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "h:?o:O:f:alq:Dd:Gr:R:cnv:",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h:?o:O:f:alq:Dd:Gr:R:cnv:W::",loptions,NULL)) >= 0)
{
switch (c) {
case 'c': args->compact_PS = 1; break;
args->verbose = strtol(optarg, &tmp, 0);
if ( *tmp || args->verbose<0 || args->verbose>1 ) error("Error: currently only --verbose 0 or --verbose 1 is supported\n");
break;
- case 13 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
if ( args->regions_list && !args->allow_overlaps ) error("The -r/-R option is supported only with -a\n");
if ( args->naive_concat )
{
+ if ( args->write_index ) error("Error: cannot --write-index in the %s mode\n",args->naive_concat_trust_headers?"--naive-force":"--naive");
if ( args->allow_overlaps ) error("The option --naive cannot be combined with --allow-overlaps\n");
if ( args->phased_concat ) error("The option --naive cannot be combined with --ligate\n");
if ( args->sites_only ) error("The option --naive cannot be combined with --drop-genotypes\n");
#include <errno.h>
#include <sys/stat.h>
#include <sys/types.h>
+#include <stdint.h>
#include <inttypes.h>
#include <htslib/faidx.h>
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/kseq.h>
+#include <htslib/hts_endian.h>
#include "bcftools.h"
#include "filter.h"
#include "convert.h"
{
long end = strtol(se+1,&ss,10);
if ( ss==se+1 ) return -1;
- bcf_update_info_int32(args->header, rec, "END", &end, 1);
+ if (end < 1 || end > INT32_MAX)
+ return -1;
+ int32_t e = end; // bcf_update_info_int32 needs an int32_t pointer
+ bcf_update_info_int32(args->header, rec, "END", &e, 1);
}
rec->rid = rid;
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
bcf1_t *rec = bcf_init();
nsamples -= 2;
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
bcf1_t *rec = bcf_init();
args->gts = (int32_t *) malloc(sizeof(int32_t)*nsamples*2);
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
bcf1_t *rec = bcf_init();
nsamples -= 2;
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
tsv_t *tsv = tsv_init(args->columns ? args->columns : "ID,CHROM,POS,AA");
if ( tsv_register(tsv, "CHROM", tsv_setter_chrom, args->header) < 0 ) error("Expected CHROM column\n");
bcf_hdr_t *hdr = bcf_sr_get_header(args->files,0);
if ( bcf_hdr_write(out_fh,hdr)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
while ( bcf_sr_next_line(args->files) )
{
bcf_hdr_t *hdr = bcf_sr_get_header(args->files,0);
if (args->record_cmd_line) bcf_hdr_append_version(hdr, args->argc, args->argv, "bcftools_convert");
if ( bcf_hdr_write(out_fh,hdr)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,hdr,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,hdr,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
int32_t *itmp = NULL, nitmp = 0;
fprintf(stderr, " -o, --output FILE Output file name [stdout]\n");
fprintf(stderr, " -O, --output-type u|b|v|z[0-9] u/b: un/compressed BCF, v/z: un/compressed VCF, 0-9: compression level [v]\n");
fprintf(stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
fprintf(stderr, "GEN/SAMPLE conversion (input/output from IMPUTE2):\n");
fprintf(stderr, " -G, --gensample2vcf ... <PREFIX>|<GEN-FILE>,<SAMPLE-FILE>\n");
{"fasta-ref",required_argument,NULL,'f'},
{"no-version",no_argument,NULL,10},
{"keep-duplicates",no_argument,NULL,12},
- {"write-index",no_argument,NULL,16},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "?h:r:R:s:S:t:T:i:e:g:G:o:O:c:f:H:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "?h:r:R:s:S:t:T:i:e:g:G:o:O:c:f:H:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'e':
if ( args->filter_str ) error("Error: only one -i or -e expression can be given, and they cannot be combined\n");
case 7 : args->convert_func = vcf_to_hapsample; args->outfname = optarg; break;
case 8 : error("The --chrom option has been deprecated, please use --3N6 instead\n"); break;
case 15 : args->gen_3N6 = 1; break;
- case 16 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'H': args->convert_func = haplegendsample_to_vcf; args->infname = optarg; break;
case 'f': args->ref_fname = optarg; break;
case 'c': args->columns = optarg; break;
#include <errno.h>
#include <sys/stat.h>
#include <sys/types.h>
+#include <stdint.h>
#include <inttypes.h>
#include <htslib/faidx.h>
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/kseq.h>
+#include <htslib/hts_endian.h>
#include "bcftools.h"
#include "filter.h"
#include "convert.h"
{
long end = strtol(se+1,&ss,10);
if ( ss==se+1 ) return -1;
- bcf_update_info_int32(args->header, rec, "END", &end, 1);
+ if (end < 1 || end > INT32_MAX)
+ return -1;
+ int32_t e = end; // bcf_update_info_int32 needs an int32_t pointer
+ bcf_update_info_int32(args->header, rec, "END", &e, 1);
}
rec->rid = rid;
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
bcf1_t *rec = bcf_init();
nsamples -= 2;
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
bcf1_t *rec = bcf_init();
args->gts = (int32_t *) malloc(sizeof(int32_t)*nsamples*2);
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
bcf1_t *rec = bcf_init();
nsamples -= 2;
if ( out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->outfname, strerror(errno));
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if ( bcf_hdr_write(out_fh,args->header)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
tsv_t *tsv = tsv_init(args->columns ? args->columns : "ID,CHROM,POS,AA");
if ( tsv_register(tsv, "CHROM", tsv_setter_chrom, args->header) < 0 ) error("Expected CHROM column\n");
bcf_hdr_t *hdr = bcf_sr_get_header(args->files,0);
if ( bcf_hdr_write(out_fh,hdr)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,args->header,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,args->header,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
while ( bcf_sr_next_line(args->files) )
{
bcf_hdr_t *hdr = bcf_sr_get_header(args->files,0);
if (args->record_cmd_line) bcf_hdr_append_version(hdr, args->argc, args->argv, "bcftools_convert");
if ( bcf_hdr_write(out_fh,hdr)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->outfname);
- if ( args->write_index && init_index(out_fh,hdr,args->outfname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->outfname);
+ if ( init_index2(out_fh,hdr,args->outfname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->outfname);
int32_t *itmp = NULL, nitmp = 0;
fprintf(bcftools_stderr, " -o, --output FILE Output file name [bcftools_stdout]\n");
fprintf(bcftools_stderr, " -O, --output-type u|b|v|z[0-9] u/b: un/compressed BCF, v/z: un/compressed VCF, 0-9: compression level [v]\n");
fprintf(bcftools_stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "GEN/SAMPLE conversion (input/output from IMPUTE2):\n");
fprintf(bcftools_stderr, " -G, --gensample2vcf ... <PREFIX>|<GEN-FILE>,<SAMPLE-FILE>\n");
{"fasta-ref",required_argument,NULL,'f'},
{"no-version",no_argument,NULL,10},
{"keep-duplicates",no_argument,NULL,12},
- {"write-index",no_argument,NULL,16},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "?h:r:R:s:S:t:T:i:e:g:G:o:O:c:f:H:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "?h:r:R:s:S:t:T:i:e:g:G:o:O:c:f:H:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'e':
if ( args->filter_str ) error("Error: only one -i or -e expression can be given, and they cannot be combined\n");
case 7 : args->convert_func = vcf_to_hapsample; args->outfname = optarg; break;
case 8 : error("The --chrom option has been deprecated, please use --3N6 instead\n"); break;
case 15 : args->gen_3N6 = 1; break;
- case 16 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'H': args->convert_func = haplegendsample_to_vcf; args->infname = optarg; break;
case 'f': args->ref_fname = optarg; break;
case 'c': args->columns = optarg; break;
fprintf(stderr, " -T, --targets-file FILE Similar to -R but streams rather than index-jumps\n");
fprintf(stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
fprintf(stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
exit(1);
}
{"SnpGap",required_argument,NULL,'g'},
{"IndelGap",required_argument,NULL,'G'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,12},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "e:i:t:T:r:R:h?s:m:M:o:O:g:G:S:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "e:i:t:T:r:R:h?s:m:M:o:O:g:G:S:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'g':
args->snp_gap = strtol(optarg,&tmp,10);
else if ( !strcasecmp(optarg,"2") ) args->mask_overlap = 2;
else error("Could not parse: --mask-overlap %s\n",optarg);
break;
- case 12 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
init_data(args);
if ( bcf_hdr_write(args->out_fh, args->hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr,args->output_fname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
while ( bcf_sr_next_line(args->files) )
{
bcf1_t *line = bcf_sr_get_line(args->files, 0);
fprintf(bcftools_stderr, " -T, --targets-file FILE Similar to -R but streams rather than index-jumps\n");
fprintf(bcftools_stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
fprintf(bcftools_stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
{"SnpGap",required_argument,NULL,'g'},
{"IndelGap",required_argument,NULL,'G'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,12},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "e:i:t:T:r:R:h?s:m:M:o:O:g:G:S:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "e:i:t:T:r:R:h?s:m:M:o:O:g:G:S:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'g':
args->snp_gap = strtol(optarg,&tmp,10);
else if ( !strcasecmp(optarg,"2") ) args->mask_overlap = 2;
else error("Could not parse: --mask-overlap %s\n",optarg);
break;
- case 12 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
init_data(args);
if ( bcf_hdr_write(args->out_fh, args->hdr)!=0 ) error("[%s] Error: cannot write the header to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr,args->output_fname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
while ( bcf_sr_next_line(args->files) )
{
bcf1_t *line = bcf_sr_get_line(args->files, 0);
/* vcfgtcheck.c -- Check sample identity.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/vcfutils.h>
#include <htslib/kbitset.h>
#include <htslib/hts_os.h>
+#include <htslib/bgzf.h>
#include <inttypes.h>
#include <sys/time.h>
#include "bcftools.h"
#include "extsort.h"
//#include "hclust.h"
+#include "filter.h"
+
+// Logic of the filters: include or exclude sites which match the filters?
+#define FLT_INCLUDE 1
+#define FLT_EXCLUDE 2
typedef struct
{
bcf_srs_t *files; // first reader is the query VCF - single sample normally or multi-sample for cross-check
bcf_hdr_t *gt_hdr, *qry_hdr; // VCF with genotypes to compare against and the query VCF
char *cwd, **argv, *gt_samples, *qry_samples, *regions, *targets, *qry_fname, *gt_fname, *pair_samples;
- int argc, gt_samples_is_file, qry_samples_is_file, regions_is_file, targets_is_file, pair_samples_is_file;
- int regions_overlap, targets_overlap;
+ char *output_fname;
+ int argc, gt_samples_is_file, qry_samples_is_file, regions_is_file, targets_is_file, pair_samples_is_file, output_type;;
+ int regions_overlap, targets_overlap, clevel;
int qry_use_GT,gt_use_GT, nqry_smpl,ngt_smpl, *qry_smpl,*gt_smpl;
int nused[2][2];
double *pdiff, *qry_prob, *gt_prob;
- uint32_t *ndiff,*ncnt,ncmp, npairs;
+ uint32_t *ndiff,*ncnt,*nmatch,ncmp, npairs;
int32_t *qry_arr,*gt_arr, nqry_arr,ngt_arr;
uint8_t *qry_dsg, *gt_dsg;
pair_t *pairs;
double *hwe_prob, dsg2prob[8][3], pl2prob[256];
double min_inter_err, max_intra_err;
- int all_sites, hom_only, ntop, cross_check, calc_hwe_prob, sort_by_hwe, dry_run, use_PLs;
- FILE *fp;
- unsigned int nskip_no_match, nskip_not_ba, nskip_mono, nskip_no_data, nskip_dip_GT, nskip_dip_PL;
+ int all_sites, hom_only, ntop, cross_check, calc_hwe_prob, sort_by_hwe, dry_run, gt_err;
+ BGZF *out_fh;
+ unsigned int nskip_no_match, nskip_not_ba, nskip_mono, nskip_no_data, nskip_dip_GT, nskip_dip_PL, nskip_filter;
+ kstring_t kstr;
// for --distinctive-sites
double distinctive_sites;
size_t diff_sites_size;
extsort_t *es;
char *es_tmp_prefix, *es_max_mem;
+
+ // include or exclude sites which match the filters
+ filter_t *qry_filter, *gt_filter;
+ char *qry_filter_str, *gt_filter_str;
+ int qry_filter_logic, gt_filter_logic; // FLT_INCLUDE or FLT_EXCLUDE
}
args_t;
}
assert(buf);
}
-static void print_header(args_t *args, FILE *fp)
+static void print_header(args_t *args)
{
- fprintf(fp, "# This file was produced by bcftools (%s+htslib-%s), the command line was:\n", bcftools_version(), hts_version());
- fprintf(fp, "# \t bcftools %s ", args->argv[0]);
+ args->kstr.l = 0;
+ ksprintf(&args->kstr, "# This file was produced by bcftools (%s+htslib-%s), the command line was:\n", bcftools_version(), hts_version());
+ ksprintf(&args->kstr, "# \t bcftools %s ", args->argv[0]);
int i;
for (i=1; i<args->argc; i++)
- fprintf(fp, " %s",args->argv[i]);
- fprintf(fp, "\n# and the working directory was:\n");
- fprintf(fp, "# \t %s\n#\n", args->cwd);
+ ksprintf(&args->kstr, " %s",args->argv[i]);
+ ksprintf(&args->kstr, "\n# and the working directory was:\n");
+ ksprintf(&args->kstr, "# \t %s\n#\n", args->cwd);
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
static int cmp_int(const void *_a, const void *_b)
typedef struct
{
- uint32_t ndiff,rid,pos,rand; // rand is to shuffle sites with the same ndiff from across all chromosoms
+ uint32_t ndiff,rid,pos,rand; // rand is to shuffle sites with the same ndiff from across all chromosomes
unsigned long kbs_dat[1];
}
diff_sites_t;
if ( !bcf_hdr_nsamples(args->gt_hdr) ) error("No samples in %s?\n", args->gt_fname);
}
+ if ( args->gt_hdr && args->gt_filter_str ) args->gt_filter = filter_init(args->gt_hdr, args->gt_filter_str);
+ if ( args->qry_hdr && args->qry_filter_str ) args->qry_filter = filter_init(args->qry_hdr, args->qry_filter_str);
+
// Determine whether GT or PL will be used
if ( args->qry_use_GT==-1 ) // not set by -u, qry uses PL by default
{
args->qry_dsg = (uint8_t*) malloc(args->nqry_smpl);
args->gt_dsg = args->cross_check ? args->qry_dsg : (uint8_t*) malloc(args->ngt_smpl);
}
- if ( args->use_PLs )
+ if ( args->gt_err )
{
args->pdiff = (double*) calloc(args->npairs,sizeof(*args->pdiff)); // log probability of pair samples being the same
args->qry_prob = (double*) malloc(3*args->nqry_smpl*sizeof(*args->qry_prob));
args->gt_prob = args->cross_check ? args->qry_prob : (double*) malloc(3*args->ngt_smpl*sizeof(*args->gt_prob));
+ // Convert genotypes to genotype likelihoods given by -E, the probability of reading one allele incorrectly. In this
+ // simple model we have:
+ // - probability of reading an allele incorrectly, eg. 0 as 1 or 1 as 0
+ // P(0|1) = P(1|0) = e
+ // - probability of genotype G={00,01,11} being correct given observed dosage {0,1,2} and the
+ // genotyping error probability e:
+ // P(00|0) = 1 P(00|1) = e P(00|2) = e^2
+ // P(01|0) = e P(01|1) = 1 P(01|2) = e
+ // P(11|0) = e^2 P(11|1) = e P(11|2) = 1
+ //
// dsg2prob: the first index is bitmask of 8 possible dsg combinations (only 1<<0,1<<2,1<<3 are set, accessing
- // anything else indicated an error, this is just to reuse gt_to_dsg()); the second index are the corresponding
+ // anything else indicated an error, this is just to reuse gt_to_dsg(); the second index are the corresponding
// probabilities of 0/0, 0/1, and 1/1 genotypes
+ //
for (i=0; i<8; i++)
for (j=0; j<3; j++)
args->dsg2prob[i][j] = HUGE_VAL;
- args->dsg2prob[1][0] = -log(1-pow(10,-0.1*args->use_PLs));
- args->dsg2prob[1][1] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[1][2] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[2][0] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[2][1] = -log(1-pow(10,-0.1*args->use_PLs));
- args->dsg2prob[2][2] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[4][0] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[4][1] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[4][2] = -log(1-pow(10,-0.1*args->use_PLs));
+ double eprob = pow(10,-0.1*args->gt_err); // convert from phred score to probability
+ args->dsg2prob[1][0] = 0; // P(00|0) = 1
+ args->dsg2prob[1][1] = -log(eprob); // P(01|0) = e
+ args->dsg2prob[1][2] = -2*log(eprob); // P(11|0) = e^2
+ args->dsg2prob[2][0] = -log(eprob); // P(00|1) = e
+ args->dsg2prob[2][1] = 0; // P(01|1) = 1
+ args->dsg2prob[2][2] = -log(eprob); // P(11|1) = e
+ args->dsg2prob[4][0] = -2*log(eprob); // P(00|2) = e^2
+ args->dsg2prob[4][1] = -log(eprob); // P(01|2) = e
+ args->dsg2prob[4][2] = 0; // P(11|2) = 1
// lookup table to avoid exponentiation
for (i=0; i<256; i++) args->pl2prob[i] = pow(10,-0.1*i);
}
else
args->ndiff = (uint32_t*) calloc(args->npairs,sizeof(*args->ndiff)); // number of differing genotypes for each pair of samples
- args->ncnt = (uint32_t*) calloc(args->npairs,sizeof(*args->ncnt)); // number of comparisons performed (non-missing data)
+ args->ncnt = (uint32_t*) calloc(args->npairs,sizeof(*args->ncnt)); // number of comparisons performed (non-missing data)
if ( !args->ncnt ) error("Error: failed to allocate %.1f Mb\n", args->npairs*sizeof(*args->ncnt)/1e6);
if ( args->calc_hwe_prob )
{
// prob of the observed sequence of matches given site AFs and HWE
args->hwe_prob = (double*) calloc(args->npairs,sizeof(*args->hwe_prob));
if ( !args->hwe_prob ) error("Error: failed to allocate %.1f Mb. Run with --no-HWE-prob to save some memory.\n", args->npairs*sizeof(*args->hwe_prob)/1e6);
+ args->nmatch = (uint32_t*) calloc(args->npairs,sizeof(*args->ncnt)); // number of matches, used only with calc_hwe_prob
+ if ( !args->nmatch ) error("Error: failed to allocate %.1f Mb.\n", args->npairs*sizeof(*args->ncnt)/1e6);
}
if ( args->distinctive_sites ) diff_sites_init(args);
- args->fp = stdout;
- print_header(args, args->fp);
+ args->out_fh = bgzf_open(args->output_fname, args->output_type&FT_GZ ? "wg" : "wu");
+ if ( args->out_fh == NULL )
+ error("[%s] Error: cannot write to %s: %s\n", __func__,args->output_fname ? args->output_fname : "standard output", strerror(errno));
+
+ print_header(args);
}
static void destroy_data(args_t *args)
{
+ free(args->kstr.s);
+ if ( args->gt_filter ) filter_destroy(args->gt_filter);
+ if ( args->qry_filter ) filter_destroy(args->qry_filter);
if ( args->gt_dsg!=args->qry_dsg ) free(args->gt_dsg);
free(args->qry_dsg);
if ( args->gt_prob!=args->qry_prob ) free(args->gt_prob);
free(args->qry_prob);
free(args->es_max_mem);
- fclose(args->fp);
+ if ( bgzf_close(args->out_fh)!=0 ) error("Error: close failed .. %s\n", args->output_fname?args->output_fname:"stdout");
if ( args->distinctive_sites ) diff_sites_destroy(args);
free(args->hwe_prob);
free(args->cwd);
free(args->pdiff);
free(args->ndiff);
free(args->ncnt);
+ free(args->nmatch);
free(args->qry_smpl);
if ( args->gt_smpl!=args->qry_smpl ) free(args->gt_smpl);
free(args->pairs);
int i,j,k, nqry1, ngt1, ret;
bcf1_t *gt_rec = NULL, *qry_rec = bcf_sr_get_line(args->files,0); // the query file
+ if ( args->qry_filter )
+ {
+ int pass = filter_test(args->qry_filter, qry_rec, NULL);
+ if ( args->qry_filter_logic==FLT_EXCLUDE ) pass = pass ? 0 : 1;
+ if ( !pass ) { args->nskip_filter++; return; }
+ }
+
int qry_use_GT = args->qry_use_GT;
int gt_use_GT = args->gt_use_GT;
if ( args->gt_hdr )
{
gt_rec = bcf_sr_get_line(args->files,1);
+ if ( args->gt_filter )
+ {
+ int pass = filter_test(args->gt_filter, gt_rec, NULL);
+ if ( args->gt_filter_logic==FLT_EXCLUDE ) pass = pass ? 0 : 1;
+ if ( !pass ) { args->nskip_filter++; return; }
+ }
ret = set_data(args, args->gt_hdr, gt_rec, &args->gt_arr, &args->ngt_arr, &ngt1, >_use_GT);
if ( ret<0 ) return;
}
args->ncmp++;
args->nused[qry_use_GT][gt_use_GT]++;
- double af,hwe_dsg[8];
+ double hwe_dsg[8];
if ( args->calc_hwe_prob )
{
int ac[2];
}
else if ( bcf_calc_ac(args->qry_hdr, qry_rec, ac, BCF_UN_INFO|BCF_UN_FMT)!=1 ) error("todo: bcf_calc_ac() failed\n");
- // hwe indexes correspond to the bitmask of eight dsg combinations to account for PL uncertainty
- // for in the extreme case we can have uninformative PL=0,0,0. So the values are the minima of e.g.
- // hwe[1,2,4] .. dsg=0,1,2
- // hwe[3] .. dsg=0 or 1
- // hwe[6] .. dsg=1 or 2
-
- double hwe[3];
- const double min_af = 1e-5; // cap the AF in case we get unrealistic values
- af = (double)ac[1]/(ac[0]+ac[1]);
- hwe[0] = af>min_af ? -log(af*af) : -log(min_af*min_af);
- hwe[1] = af>min_af && af<1-min_af ? -log(2*af*(1-af)) : -log(2*min_af*(1-min_af));
- hwe[2] = af<(1-min_af) ? -log((1-af)*(1-af)) : -log(min_af*min_af);
+ // Calculate HWE probability for each possible qry+gt dosage combination. The alternate allele dosage
+ // values returned by gt_to_prob() below are 0,1,2,4 (0=missing, 1<<0, 1<<1, 1<<2). We consider only
+ // biallelic sites, therefore we work with eight genotype combinations.
+ //
+ // The array hwe_dsg is accessed with hwe_dsg[qry_dsg & gt_dsg] and is constructed to account for PL uncertainty
+ // when we encounter less informative PL, such as PL=0,0,10, where multiple dosage values are equally
+ // likely. If we allowed complete uncertainty (PL=0,0,0), we'd have up to eight possible genotype
+ // mask combinations: from e.g. 0=(gt_dsg=1<<0 & qry_dsg=1<<1) to 7=(gt_dsg=1<<0|1<<1|1<<2 & qry_dsg=1<<0|1<<1|1<<2).
+ // Note the extreme case of 1|2|4 is skipped, see pl_to_dsg().
+ //
+ // When the dosage is uncertain, we take the minimum of their corresponding HWE value, for example
+ // hwe[0] = 0
+ // hwe[1] = (1-AF)**2
+ // hwe[2] = 2*AF*(1-AF)
+ // hwe[4] = AF**2
+ // hwe[3] = min{hwe[1],hwe[2]}
+
+ double hwe[3]; // while hwe_dsg iterates over dsg bitmasks (0..7), hwe iterates over dsg (0,1,2)
+ double af = ac[0]+ac[1] ? (double)ac[1]/(ac[0]+ac[1]) : 1e-6;
+ hwe[0] = -log((1-af)*(1-af));
+ hwe[1] = -log(2*af*(1-af));
+ hwe[2] = -log(af*af);
hwe_dsg[0] = 0;
for (i=1; i<8; i++)
{
// The sample pairs were given explicitly via -p/-P options
if ( args->pairs )
{
- if ( !args->use_PLs )
+ if ( !args->gt_err )
{
int ndiff = 0;
if ( args->kbs_diff ) diff_sites_reset(args);
args->ndiff[i]++;
if ( args->kbs_diff ) { ndiff++; kbs_insert(args->kbs_diff, i); }
}
- else if ( args->calc_hwe_prob ) args->hwe_prob[i] += hwe_dsg[match];
+ else if ( args->calc_hwe_prob )
+ {
+ args->hwe_prob[i] += hwe_dsg[match];
+ args->nmatch[i]++;
+ }
args->ncnt[i]++;
}
if ( ndiff ) diff_sites_push(args, ndiff, qry_rec->rid, qry_rec->pos);
}
- else // use_PLs set
+ else // gt_err set
{
for (i=0; i<args->npairs; i++)
{
{
int match = qry_dsg & gt_dsg;
args->hwe_prob[i] += hwe_dsg[match];
+ if ( match ) args->nmatch[i]++;
}
args->ncnt[i]++;
}
}
int idx=0;
- if ( !args->use_PLs )
+ if ( !args->gt_err )
{
for (i=0; i<args->nqry_smpl; i++)
{
if ( !args->gt_dsg[j] ) { idx++; continue; } // missing value
int match = args->qry_dsg[i] & args->gt_dsg[j];
if ( !match ) args->ndiff[idx]++;
- else if ( args->calc_hwe_prob ) args->hwe_prob[idx] += hwe_dsg[match];
+ else if ( args->calc_hwe_prob )
+ {
+ args->hwe_prob[idx] += hwe_dsg[match];
+ args->nmatch[idx]++;
+ }
args->ncnt[idx]++;
idx++;
}
}
}
- else // use_PLs set
+ else // gt_err set
{
for (i=0; i<args->nqry_smpl; i++)
{
{
int match = args->qry_dsg[i] & args->gt_dsg[j];
args->hwe_prob[idx] += hwe_dsg[match];
+ if ( match ) args->nmatch[idx]++;
}
args->ncnt[idx]++;
idx++;
{
extsort_sort(args->es);
- fprintf(args->fp,"# DS, distinctive sites:\n");
- fprintf(args->fp,"# - chromosome\n");
- fprintf(args->fp,"# - position\n");
- fprintf(args->fp,"# - cumulative number of pairs distinguished by this block\n");
- fprintf(args->fp,"# - block id\n");
- fprintf(args->fp,"#DS\t[2]Chromosome\t[3]Position\t[4]Cumulative number of distinct pairs\t[5]Block id\n");
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"# DS, distinctive sites:\n");
+ ksprintf(&args->kstr,"# - chromosome\n");
+ ksprintf(&args->kstr,"# - position\n");
+ ksprintf(&args->kstr,"# - cumulative number of pairs distinguished by this block\n");
+ ksprintf(&args->kstr,"# - block id\n");
+ ksprintf(&args->kstr,"#DS\t[2]Chromosome\t[3]Position\t[4]Cumulative number of distinct pairs\t[5]Block id\n");
kbitset_t *kbs_blk = kbs_init(args->npairs);
kbitset_iter_t itr;
if ( ndiff_dbg!=ndiff ) error("Corrupted data, fixme: %d vs %d\n",ndiff_dbg,ndiff);
if ( !ndiff_new ) continue; // no new pair distinguished by this site
ndiff_tot += ndiff_new;
- fprintf(args->fp,"DS\t%s\t%d\t%d\t%d\n",bcf_hdr_id2name(args->qry_hdr,rid),pos+1,ndiff_tot,iblock);
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"DS\t%s\t%d\t%d\t%d\n",bcf_hdr_id2name(args->qry_hdr,rid),pos+1,ndiff_tot,iblock);
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
if ( ndiff_tot < ndiff_min ) continue; // fewer than the requested number of pairs can be distinguished at this point
iblock++;
ndiff_tot = 0;
}
static void report(args_t *args)
{
- fprintf(args->fp,"INFO\tsites-compared\t%u\n",args->ncmp);
- fprintf(args->fp,"INFO\tsites-skipped-no-match\t%u\n",args->nskip_no_match);
- fprintf(args->fp,"INFO\tsites-skipped-multiallelic\t%u\n",args->nskip_not_ba);
- fprintf(args->fp,"INFO\tsites-skipped-monoallelic\t%u\n",args->nskip_mono);
- fprintf(args->fp,"INFO\tsites-skipped-no-data\t%u\n",args->nskip_no_data);
- fprintf(args->fp,"INFO\tsites-skipped-GT-not-diploid\t%u\n",args->nskip_dip_GT);
- fprintf(args->fp,"INFO\tsites-skipped-PL-not-diploid\t%u\n",args->nskip_dip_PL);
- fprintf(args->fp,"INFO\tsites-used-PL-vs-PL\t%u\n",args->nused[0][0]);
- fprintf(args->fp,"INFO\tsites-used-PL-vs-GT\t%u\n",args->nused[0][1]);
- fprintf(args->fp,"INFO\tsites-used-GT-vs-PL\t%u\n",args->nused[1][0]);
- fprintf(args->fp,"INFO\tsites-used-GT-vs-GT\t%u\n",args->nused[1][1]);
- fprintf(args->fp,"# DC, discordance:\n");
- fprintf(args->fp,"# - query sample\n");
- fprintf(args->fp,"# - genotyped sample\n");
- fprintf(args->fp,"# - discordance (either an abstract score or number of mismatches, see -e/-u in the man page for details; smaller is better)\n");
- fprintf(args->fp,"# - negative log of HWE probability at matching sites (rare genotypes matches are more informative, bigger is better)\n");
- fprintf(args->fp,"# - number of sites compared (bigger is better)\n");
- fprintf(args->fp,"#DC\t[2]Query Sample\t[3]Genotyped Sample\t[4]Discordance\t[5]-log P(HWE)\t[6]Number of sites compared\n");
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"INFO\tsites-compared\t%u\n",args->ncmp);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-no-match\t%u\n",args->nskip_no_match);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-multiallelic\t%u\n",args->nskip_not_ba);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-monoallelic\t%u\n",args->nskip_mono);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-no-data\t%u\n",args->nskip_no_data);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-GT-not-diploid\t%u\n",args->nskip_dip_GT);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-PL-not-diploid\t%u\n",args->nskip_dip_PL);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-filtering-expression\t%u\n",args->nskip_filter);
+ ksprintf(&args->kstr,"INFO\tsites-used-PL-vs-PL\t%u\n",args->nused[0][0]);
+ ksprintf(&args->kstr,"INFO\tsites-used-PL-vs-GT\t%u\n",args->nused[0][1]);
+ ksprintf(&args->kstr,"INFO\tsites-used-GT-vs-PL\t%u\n",args->nused[1][0]);
+ ksprintf(&args->kstr,"INFO\tsites-used-GT-vs-GT\t%u\n",args->nused[1][1]);
+ ksprintf(&args->kstr,"# DCv2, discordance version 2:\n");
+ ksprintf(&args->kstr,"# - Query sample\n");
+ ksprintf(&args->kstr,"# - Genotyped sample\n");
+ ksprintf(&args->kstr,"# - Discordance, given either as an abstract score or number of mismatches, see the options -E/-u\n"
+ "# in man page for details. Note that samples with high missingness have fewer sites compared,\n"
+ "# which results in lower overall discordance. Therefore it is advisable to use the average score\n"
+ "# per site rather than the absolute value, i.e. divide the value by the number of sites compared\n"
+ "# (smaller value = better match)\n");
+ ksprintf(&args->kstr,"# - Average negative log of HWE probability at matching sites, attempts to quantify the following\n"
+ "# intuition: rare genotype matches are more informative than common genotype matches, hence two\n"
+ "# samples with similar discordance can be further stratified by the HWE score (bigger value = better\n"
+ "# match, the observed concordance was less likely to occur by chance)\n");
+ ksprintf(&args->kstr,"# - Number of sites compared for this pair of samples (bigger = more informative)\n");
+ ksprintf(&args->kstr,"# - Number of matching genotypes\n");
+ ksprintf(&args->kstr,"#DCv2\t[2]Query Sample\t[3]Genotyped Sample\t[4]Discordance\t[5]Average -log P(HWE)\t[6]Number of sites compared\t[7]Number of matching genotypes\n");
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
int trim = args->ntop;
if ( !args->pairs )
int i;
for (i=0; i<args->npairs; i++)
{
+ args->kstr.l = 0;
int iqry = args->pairs[i].iqry;
int igt = args->pairs[i].igt;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->ndiff[i],
- args->calc_hwe_prob ? args->hwe_prob[i] : 0,
- args->ncnt[i]);
+ (args->calc_hwe_prob && args->nmatch[i]) ? args->hwe_prob[i]/args->nmatch[i] : 0,
+ args->ncnt[i],
+ args->nmatch[i]);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->pdiff[i],
- args->calc_hwe_prob ? args->hwe_prob[i] : 0,
- args->ncnt[i]);
+ (args->calc_hwe_prob && args->nmatch[i]) ? args->hwe_prob[i]/args->nmatch[i] : 0,
+ args->ncnt[i],
+ args->nmatch[i]);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
}
else if ( !trim )
int ngt = args->cross_check ? i : args->ngt_smpl;
for (j=0; j<ngt; j++)
{
+ args->kstr.l = 0;
int igt = args->gt_smpl ? args->gt_smpl[j] : j;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->ndiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->pdiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
idx++;
}
}
for (j=0; j<args->ngt_smpl; j++)
{
if ( args->sort_by_hwe )
- arr[j].val = -args->hwe_prob[idx];
+ arr[j].val = args->nmatch[idx] ? -args->hwe_prob[idx]/args->nmatch[idx] : 0; // -args->hwe_prob[idx];
else if ( args->ndiff )
arr[j].val = args->ncnt[idx] ? (double)args->ndiff[idx]/args->ncnt[idx] : 0;
else
int iqry = args->qry_smpl ? args->qry_smpl[i] : i;
for (j=0; j<args->ntop; j++)
{
+ args->kstr.l = 0;
int idx = arr[j].idx;
int igt = args->gt_smpl ? args->gt_smpl[arr[j].ism] : arr[j].ism;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->ndiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->pdiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
}
free(arr);
for (j=0; j<i; j++)
{
if ( args->sort_by_hwe )
- arr[k].val = -args->hwe_prob[idx];
+ arr[k].val = args->nmatch[idx] ? -args->hwe_prob[idx]/args->nmatch[idx] : 0;
else if ( args->ndiff )
arr[k].val = args->ncnt[idx] ? (double)args->ndiff[idx]/args->ncnt[idx] : 0;
else
{
idx = j*(j+1)/2 + i;
if ( args->sort_by_hwe )
- arr[k].val = -args->hwe_prob[idx];
+ arr[k].val = args->nmatch[idx] ? -args->hwe_prob[idx]/args->nmatch[idx] : 0;
else if ( args->ndiff )
arr[k].val = args->ncnt[idx] ? (double)args->ndiff[idx]/args->ncnt[idx] : 0;
else
int iqry = args->qry_smpl ? args->qry_smpl[i] : i;
for (j=0; j<args->ntop; j++)
{
+ args->kstr.l = 0;
if ( i <= arr[j].ism ) continue;
int idx = arr[j].idx;
int igt = args->qry_smpl ? args->qry_smpl[arr[j].ism] : arr[j].ism;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->qry_hdr->samples[igt],
args->ndiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->qry_hdr->samples[igt],
args->pdiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
}
free(arr);
fprintf(stderr, " and TMP is a prefix of temporary files used by external sorting [/tmp/bcftools.XXXXXX]\n");
#endif
fprintf(stderr, " --dry-run Stop after first record to estimate required time\n");
- fprintf(stderr, " -e, --error-probability INT Phred-scaled probability of genotyping error, 0 for faster but less accurate results [40]\n");
+ fprintf(stderr, " -E, --error-probability INT Phred-scaled probability of genotyping error, 0 for faster but less accurate results [40]\n");
+ fprintf(stderr, " -e, --exclude [qry|gt]:EXPR Exclude sites for which the expression is true\n");
fprintf(stderr, " -g, --genotypes FILE Genotypes to compare against\n");
fprintf(stderr, " -H, --homs-only Homozygous genotypes only, useful with low coverage data (requires -g)\n");
+ fprintf(stderr, " -i, --include [qry|gt]:EXPR Include sites for which the expression is true\n");
fprintf(stderr, " --n-matches INT Print only top INT matches for each sample (sorted by average score), 0 for unlimited.\n");
fprintf(stderr, " Use negative value to sort by HWE probability rather than by discordance [0]\n");
fprintf(stderr, " --no-HWE-prob Disable calculation of HWE probability\n");
+ fprintf(stderr, " -o, --output FILE Write output to a file [standard output]\n");
+ fprintf(stderr, " -O, --output-type t|z t: plain tab-delimited text output, z: compressed [t]\n");
fprintf(stderr, " -p, --pairs LIST Comma-separated sample pairs to compare (qry,gt[,qry,gt..] with -g or qry,qry[,qry,qry..] w/o)\n");
fprintf(stderr, " -P, --pairs-file FILE File with tab-delimited sample pairs to compare (qry,gt with -g or qry,qry w/o)\n");
fprintf(stderr, " -r, --regions REGION Restrict to comma-separated list of regions\n");
fprintf(stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
fprintf(stderr, " -u, --use TAG1[,TAG2] Which tag to use in the query file (TAG1) and the -g file (TAG2) [PL,GT]\n");
fprintf(stderr, "Examples:\n");
- fprintf(stderr, " # Check discordance of all samples from B against all sample in A\n");
+ fprintf(stderr, " # Check discordance of all samples from B against all samples in A\n");
fprintf(stderr, " bcftools gtcheck -g A.bcf B.bcf\n");
fprintf(stderr, "\n");
- fprintf(stderr, " # Limit comparisons to the fiven list of samples\n");
+ fprintf(stderr, " # Limit comparisons to the given list of samples\n");
fprintf(stderr, " bcftools gtcheck -s gt:a1,a2,a3 -s qry:b1,b2 -g A.bcf B.bcf\n");
fprintf(stderr, "\n");
fprintf(stderr, " # Compare only two pairs a1,b1 and a1,b2\n");
args->qry_use_GT = -1;
args->gt_use_GT = -1;
args->calc_hwe_prob = 1;
- args->use_PLs = 40;
+ args->gt_err = 40;
args->regions_overlap = 1;
args->targets_overlap = 0;
+ args->output_fname = "-";
// external sort for --distinctive-sites
#ifdef _WIN32
static struct option loptions[] =
{
- {"error-probability",1,0,'e'},
+ {"error-probability",1,0,'E'}, // note this used to be 'e', but can easily auto-detect to assure backward compatibility
+ {"exclude",required_argument,0,'e'},
+ {"include",required_argument,0,'i'},
+ {"output",required_argument,0,'o'},
+ {"output-type",required_argument,NULL,'O'},
{"use",1,0,'u'},
{"cluster",1,0,'c'},
{"GTs-only",1,0,'G'},
{0,0,0,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hg:p:s:S:p:P:Hr:R:at:T:G:c:u:e:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hg:p:s:S:p:P:Hr:R:at:T:G:c:u:e:E:i:o:O:",loptions,NULL)) >= 0) {
switch (c) {
+ case 'o': args->output_fname = optarg; break;
+ case 'O':
+ switch (optarg[0]) {
+ case 't': args->output_type = FT_TAB_TEXT; break;
+ case 'z': args->output_type = FT_VCF_GZ; break;
+ default:
+ {
+ args->clevel = strtol(optarg,&tmp,10);
+ if ( *tmp || args->clevel<0 || args->clevel>9 ) error("The output type \"%s\" not recognised\n", optarg);
+ }
+ }
+ if ( optarg[1] )
+ {
+ args->clevel = strtol(optarg+1,&tmp,10);
+ if ( *tmp || args->clevel<0 || args->clevel>9 ) error("Could not parse argument: --output-type %s\n", optarg+1);
+ }
+ break;
case 'e':
- args->use_PLs = strtol(optarg,&tmp,10);
+ if ( !strncasecmp("gt:",optarg,3) )
+ {
+ if ( args->gt_filter_str ) error("Error: only one -i or -e expression can be given for gt:, and they cannot be combined\n");
+ args->gt_filter_str = optarg;
+ args->gt_filter_logic |= FLT_EXCLUDE;
+ }
+ else if ( !strncasecmp("qry:",optarg,4) )
+ {
+ if ( args->qry_filter_str ) error("Error: only one -i or -e expression can be given for qry:, and they cannot be combined\n");
+ args->qry_filter_str = optarg;
+ args->qry_filter_logic |= FLT_EXCLUDE;
+ }
+ else
+ {
+ // this could be the old -e, --error-probability option
+ args->gt_err = strtol(optarg,&tmp,10);
+ if ( !tmp || *tmp )
+ {
+ // it is not
+ args->gt_filter_str = optarg;
+ args->qry_filter_str = optarg;
+ args->gt_filter_logic |= FLT_EXCLUDE;
+ args->qry_filter_logic |= FLT_EXCLUDE;
+ }
+ else
+ {
+ fprintf(stderr,"[warning] auto-detected the old format --error-probability option, please switch from -e to -E.\n");
+ }
+ }
+ break;
+ case 'i':
+ if ( !strncasecmp("gt:",optarg,3) )
+ {
+ if ( args->gt_filter_str ) error("Error: only one -i or -e expression can be given for gt:, and they cannot be combined\n");
+ args->gt_filter_str = optarg;
+ args->gt_filter_logic |= FLT_INCLUDE;
+ }
+ else if ( !strncasecmp("qry:",optarg,4) )
+ {
+ if ( args->qry_filter_str ) error("Error: only one -i or -e expression can be given for qry:, and they cannot be combined\n");
+ args->qry_filter_str = optarg;
+ args->qry_filter_logic |= FLT_INCLUDE;
+ }
+ else
+ {
+ args->gt_filter_str = optarg;
+ args->qry_filter_str = optarg;
+ args->gt_filter_logic |= FLT_INCLUDE;
+ args->qry_filter_logic |= FLT_INCLUDE;
+ }
+ break;
+ case 'E':
+ args->gt_err = strtol(optarg,&tmp,10);
if ( !tmp || *tmp ) error("Could not parse: --error-probability %s\n", optarg);
break;
case 'u':
while ( *tmp && *tmp!=',' ) tmp++;
if ( *tmp ) { *tmp = 0; args->es_tmp_prefix = tmp+1; }
}
- args->use_PLs = 0;
+ args->gt_err = 0;
break;
case 'c':
error("The -c option is to be implemented, please open an issue on github\n");
}
if ( args->distinctive_sites && !args->pair_samples ) error("The experimental option --distinctive-sites requires -p/-P\n");
if ( args->hom_only && !args->gt_fname ) error("The option --homs-only requires --genotypes\n");
- if ( args->distinctive_sites && args->use_PLs ) error("The option --distinctive-sites cannot be combined with --error-probability\n");
+ if ( args->distinctive_sites && args->gt_err ) error("The option --distinctive-sites cannot be combined with --error-probability\n");
init_data(args);
gettimeofday(&t1, NULL);
double delta = (t1.tv_sec - t0.tv_sec) * 1e6 + (t1.tv_usec - t0.tv_usec);
fprintf(stderr,"INFO:\tTime required to process one record .. %f seconds\n",delta/1e6);
- fprintf(args->fp,"INFO\tTime required to process one record .. %f seconds\n",delta/1e6);
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"INFO\tTime required to process one record .. %f seconds\n",delta/1e6);
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
if ( args->dry_run ) break;
}
}
/* vcfgtcheck.c -- Check sample identity.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/vcfutils.h>
#include <htslib/kbitset.h>
#include <htslib/hts_os.h>
+#include <htslib/bgzf.h>
#include <inttypes.h>
#include <sys/time.h>
#include "bcftools.h"
#include "extsort.h"
//#include "hclust.h"
+#include "filter.h"
+
+// Logic of the filters: include or exclude sites which match the filters?
+#define FLT_INCLUDE 1
+#define FLT_EXCLUDE 2
typedef struct
{
bcf_srs_t *files; // first reader is the query VCF - single sample normally or multi-sample for cross-check
bcf_hdr_t *gt_hdr, *qry_hdr; // VCF with genotypes to compare against and the query VCF
char *cwd, **argv, *gt_samples, *qry_samples, *regions, *targets, *qry_fname, *gt_fname, *pair_samples;
- int argc, gt_samples_is_file, qry_samples_is_file, regions_is_file, targets_is_file, pair_samples_is_file;
- int regions_overlap, targets_overlap;
+ char *output_fname;
+ int argc, gt_samples_is_file, qry_samples_is_file, regions_is_file, targets_is_file, pair_samples_is_file, output_type;;
+ int regions_overlap, targets_overlap, clevel;
int qry_use_GT,gt_use_GT, nqry_smpl,ngt_smpl, *qry_smpl,*gt_smpl;
int nused[2][2];
double *pdiff, *qry_prob, *gt_prob;
- uint32_t *ndiff,*ncnt,ncmp, npairs;
+ uint32_t *ndiff,*ncnt,*nmatch,ncmp, npairs;
int32_t *qry_arr,*gt_arr, nqry_arr,ngt_arr;
uint8_t *qry_dsg, *gt_dsg;
pair_t *pairs;
double *hwe_prob, dsg2prob[8][3], pl2prob[256];
double min_inter_err, max_intra_err;
- int all_sites, hom_only, ntop, cross_check, calc_hwe_prob, sort_by_hwe, dry_run, use_PLs;
- FILE *fp;
- unsigned int nskip_no_match, nskip_not_ba, nskip_mono, nskip_no_data, nskip_dip_GT, nskip_dip_PL;
+ int all_sites, hom_only, ntop, cross_check, calc_hwe_prob, sort_by_hwe, dry_run, gt_err;
+ BGZF *out_fh;
+ unsigned int nskip_no_match, nskip_not_ba, nskip_mono, nskip_no_data, nskip_dip_GT, nskip_dip_PL, nskip_filter;
+ kstring_t kstr;
// for --distinctive-sites
double distinctive_sites;
size_t diff_sites_size;
extsort_t *es;
char *es_tmp_prefix, *es_max_mem;
+
+ // include or exclude sites which match the filters
+ filter_t *qry_filter, *gt_filter;
+ char *qry_filter_str, *gt_filter_str;
+ int qry_filter_logic, gt_filter_logic; // FLT_INCLUDE or FLT_EXCLUDE
}
args_t;
}
assert(buf);
}
-static void print_header(args_t *args, FILE *fp)
+static void print_header(args_t *args)
{
- fprintf(fp, "# This file was produced by bcftools (%s+htslib-%s), the command line was:\n", bcftools_version(), hts_version());
- fprintf(fp, "# \t bcftools %s ", args->argv[0]);
+ args->kstr.l = 0;
+ ksprintf(&args->kstr, "# This file was produced by bcftools (%s+htslib-%s), the command line was:\n", bcftools_version(), hts_version());
+ ksprintf(&args->kstr, "# \t bcftools %s ", args->argv[0]);
int i;
for (i=1; i<args->argc; i++)
- fprintf(fp, " %s",args->argv[i]);
- fprintf(fp, "\n# and the working directory was:\n");
- fprintf(fp, "# \t %s\n#\n", args->cwd);
+ ksprintf(&args->kstr, " %s",args->argv[i]);
+ ksprintf(&args->kstr, "\n# and the working directory was:\n");
+ ksprintf(&args->kstr, "# \t %s\n#\n", args->cwd);
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
static int cmp_int(const void *_a, const void *_b)
typedef struct
{
- uint32_t ndiff,rid,pos,rand; // rand is to shuffle sites with the same ndiff from across all chromosoms
+ uint32_t ndiff,rid,pos,rand; // rand is to shuffle sites with the same ndiff from across all chromosomes
unsigned long kbs_dat[1];
}
diff_sites_t;
if ( !bcf_hdr_nsamples(args->gt_hdr) ) error("No samples in %s?\n", args->gt_fname);
}
+ if ( args->gt_hdr && args->gt_filter_str ) args->gt_filter = filter_init(args->gt_hdr, args->gt_filter_str);
+ if ( args->qry_hdr && args->qry_filter_str ) args->qry_filter = filter_init(args->qry_hdr, args->qry_filter_str);
+
// Determine whether GT or PL will be used
if ( args->qry_use_GT==-1 ) // not set by -u, qry uses PL by default
{
args->qry_dsg = (uint8_t*) malloc(args->nqry_smpl);
args->gt_dsg = args->cross_check ? args->qry_dsg : (uint8_t*) malloc(args->ngt_smpl);
}
- if ( args->use_PLs )
+ if ( args->gt_err )
{
args->pdiff = (double*) calloc(args->npairs,sizeof(*args->pdiff)); // log probability of pair samples being the same
args->qry_prob = (double*) malloc(3*args->nqry_smpl*sizeof(*args->qry_prob));
args->gt_prob = args->cross_check ? args->qry_prob : (double*) malloc(3*args->ngt_smpl*sizeof(*args->gt_prob));
+ // Convert genotypes to genotype likelihoods given by -E, the probability of reading one allele incorrectly. In this
+ // simple model we have:
+ // - probability of reading an allele incorrectly, eg. 0 as 1 or 1 as 0
+ // P(0|1) = P(1|0) = e
+ // - probability of genotype G={00,01,11} being correct given observed dosage {0,1,2} and the
+ // genotyping error probability e:
+ // P(00|0) = 1 P(00|1) = e P(00|2) = e^2
+ // P(01|0) = e P(01|1) = 1 P(01|2) = e
+ // P(11|0) = e^2 P(11|1) = e P(11|2) = 1
+ //
// dsg2prob: the first index is bitmask of 8 possible dsg combinations (only 1<<0,1<<2,1<<3 are set, accessing
- // anything else indicated an error, this is just to reuse gt_to_dsg()); the second index are the corresponding
+ // anything else indicated an error, this is just to reuse gt_to_dsg(); the second index are the corresponding
// probabilities of 0/0, 0/1, and 1/1 genotypes
+ //
for (i=0; i<8; i++)
for (j=0; j<3; j++)
args->dsg2prob[i][j] = HUGE_VAL;
- args->dsg2prob[1][0] = -log(1-pow(10,-0.1*args->use_PLs));
- args->dsg2prob[1][1] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[1][2] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[2][0] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[2][1] = -log(1-pow(10,-0.1*args->use_PLs));
- args->dsg2prob[2][2] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[4][0] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[4][1] = -log(0.5*pow(10,-0.1*args->use_PLs));
- args->dsg2prob[4][2] = -log(1-pow(10,-0.1*args->use_PLs));
+ double eprob = pow(10,-0.1*args->gt_err); // convert from phred score to probability
+ args->dsg2prob[1][0] = 0; // P(00|0) = 1
+ args->dsg2prob[1][1] = -log(eprob); // P(01|0) = e
+ args->dsg2prob[1][2] = -2*log(eprob); // P(11|0) = e^2
+ args->dsg2prob[2][0] = -log(eprob); // P(00|1) = e
+ args->dsg2prob[2][1] = 0; // P(01|1) = 1
+ args->dsg2prob[2][2] = -log(eprob); // P(11|1) = e
+ args->dsg2prob[4][0] = -2*log(eprob); // P(00|2) = e^2
+ args->dsg2prob[4][1] = -log(eprob); // P(01|2) = e
+ args->dsg2prob[4][2] = 0; // P(11|2) = 1
// lookup table to avoid exponentiation
for (i=0; i<256; i++) args->pl2prob[i] = pow(10,-0.1*i);
}
else
args->ndiff = (uint32_t*) calloc(args->npairs,sizeof(*args->ndiff)); // number of differing genotypes for each pair of samples
- args->ncnt = (uint32_t*) calloc(args->npairs,sizeof(*args->ncnt)); // number of comparisons performed (non-missing data)
+ args->ncnt = (uint32_t*) calloc(args->npairs,sizeof(*args->ncnt)); // number of comparisons performed (non-missing data)
if ( !args->ncnt ) error("Error: failed to allocate %.1f Mb\n", args->npairs*sizeof(*args->ncnt)/1e6);
if ( args->calc_hwe_prob )
{
// prob of the observed sequence of matches given site AFs and HWE
args->hwe_prob = (double*) calloc(args->npairs,sizeof(*args->hwe_prob));
if ( !args->hwe_prob ) error("Error: failed to allocate %.1f Mb. Run with --no-HWE-prob to save some memory.\n", args->npairs*sizeof(*args->hwe_prob)/1e6);
+ args->nmatch = (uint32_t*) calloc(args->npairs,sizeof(*args->ncnt)); // number of matches, used only with calc_hwe_prob
+ if ( !args->nmatch ) error("Error: failed to allocate %.1f Mb.\n", args->npairs*sizeof(*args->ncnt)/1e6);
}
if ( args->distinctive_sites ) diff_sites_init(args);
- args->fp = bcftools_stdout;
- print_header(args, args->fp);
+ args->out_fh = bgzf_open(args->output_fname, args->output_type&FT_GZ ? "wg" : "wu");
+ if ( args->out_fh == NULL )
+ error("[%s] Error: cannot write to %s: %s\n", __func__,args->output_fname ? args->output_fname : "standard output", strerror(errno));
+
+ print_header(args);
}
static void destroy_data(args_t *args)
{
+ free(args->kstr.s);
+ if ( args->gt_filter ) filter_destroy(args->gt_filter);
+ if ( args->qry_filter ) filter_destroy(args->qry_filter);
if ( args->gt_dsg!=args->qry_dsg ) free(args->gt_dsg);
free(args->qry_dsg);
if ( args->gt_prob!=args->qry_prob ) free(args->gt_prob);
free(args->qry_prob);
free(args->es_max_mem);
- fclose(args->fp);
+ if ( bgzf_close(args->out_fh)!=0 ) error("Error: close failed .. %s\n", args->output_fname?args->output_fname:"bcftools_stdout");
if ( args->distinctive_sites ) diff_sites_destroy(args);
free(args->hwe_prob);
free(args->cwd);
free(args->pdiff);
free(args->ndiff);
free(args->ncnt);
+ free(args->nmatch);
free(args->qry_smpl);
if ( args->gt_smpl!=args->qry_smpl ) free(args->gt_smpl);
free(args->pairs);
int i,j,k, nqry1, ngt1, ret;
bcf1_t *gt_rec = NULL, *qry_rec = bcf_sr_get_line(args->files,0); // the query file
+ if ( args->qry_filter )
+ {
+ int pass = filter_test(args->qry_filter, qry_rec, NULL);
+ if ( args->qry_filter_logic==FLT_EXCLUDE ) pass = pass ? 0 : 1;
+ if ( !pass ) { args->nskip_filter++; return; }
+ }
+
int qry_use_GT = args->qry_use_GT;
int gt_use_GT = args->gt_use_GT;
if ( args->gt_hdr )
{
gt_rec = bcf_sr_get_line(args->files,1);
+ if ( args->gt_filter )
+ {
+ int pass = filter_test(args->gt_filter, gt_rec, NULL);
+ if ( args->gt_filter_logic==FLT_EXCLUDE ) pass = pass ? 0 : 1;
+ if ( !pass ) { args->nskip_filter++; return; }
+ }
ret = set_data(args, args->gt_hdr, gt_rec, &args->gt_arr, &args->ngt_arr, &ngt1, >_use_GT);
if ( ret<0 ) return;
}
args->ncmp++;
args->nused[qry_use_GT][gt_use_GT]++;
- double af,hwe_dsg[8];
+ double hwe_dsg[8];
if ( args->calc_hwe_prob )
{
int ac[2];
}
else if ( bcf_calc_ac(args->qry_hdr, qry_rec, ac, BCF_UN_INFO|BCF_UN_FMT)!=1 ) error("todo: bcf_calc_ac() failed\n");
- // hwe indexes correspond to the bitmask of eight dsg combinations to account for PL uncertainty
- // for in the extreme case we can have uninformative PL=0,0,0. So the values are the minima of e.g.
- // hwe[1,2,4] .. dsg=0,1,2
- // hwe[3] .. dsg=0 or 1
- // hwe[6] .. dsg=1 or 2
-
- double hwe[3];
- const double min_af = 1e-5; // cap the AF in case we get unrealistic values
- af = (double)ac[1]/(ac[0]+ac[1]);
- hwe[0] = af>min_af ? -log(af*af) : -log(min_af*min_af);
- hwe[1] = af>min_af && af<1-min_af ? -log(2*af*(1-af)) : -log(2*min_af*(1-min_af));
- hwe[2] = af<(1-min_af) ? -log((1-af)*(1-af)) : -log(min_af*min_af);
+ // Calculate HWE probability for each possible qry+gt dosage combination. The alternate allele dosage
+ // values returned by gt_to_prob() below are 0,1,2,4 (0=missing, 1<<0, 1<<1, 1<<2). We consider only
+ // biallelic sites, therefore we work with eight genotype combinations.
+ //
+ // The array hwe_dsg is accessed with hwe_dsg[qry_dsg & gt_dsg] and is constructed to account for PL uncertainty
+ // when we encounter less informative PL, such as PL=0,0,10, where multiple dosage values are equally
+ // likely. If we allowed complete uncertainty (PL=0,0,0), we'd have up to eight possible genotype
+ // mask combinations: from e.g. 0=(gt_dsg=1<<0 & qry_dsg=1<<1) to 7=(gt_dsg=1<<0|1<<1|1<<2 & qry_dsg=1<<0|1<<1|1<<2).
+ // Note the extreme case of 1|2|4 is skipped, see pl_to_dsg().
+ //
+ // When the dosage is uncertain, we take the minimum of their corresponding HWE value, for example
+ // hwe[0] = 0
+ // hwe[1] = (1-AF)**2
+ // hwe[2] = 2*AF*(1-AF)
+ // hwe[4] = AF**2
+ // hwe[3] = min{hwe[1],hwe[2]}
+
+ double hwe[3]; // while hwe_dsg iterates over dsg bitmasks (0..7), hwe iterates over dsg (0,1,2)
+ double af = ac[0]+ac[1] ? (double)ac[1]/(ac[0]+ac[1]) : 1e-6;
+ hwe[0] = -log((1-af)*(1-af));
+ hwe[1] = -log(2*af*(1-af));
+ hwe[2] = -log(af*af);
hwe_dsg[0] = 0;
for (i=1; i<8; i++)
{
// The sample pairs were given explicitly via -p/-P options
if ( args->pairs )
{
- if ( !args->use_PLs )
+ if ( !args->gt_err )
{
int ndiff = 0;
if ( args->kbs_diff ) diff_sites_reset(args);
args->ndiff[i]++;
if ( args->kbs_diff ) { ndiff++; kbs_insert(args->kbs_diff, i); }
}
- else if ( args->calc_hwe_prob ) args->hwe_prob[i] += hwe_dsg[match];
+ else if ( args->calc_hwe_prob )
+ {
+ args->hwe_prob[i] += hwe_dsg[match];
+ args->nmatch[i]++;
+ }
args->ncnt[i]++;
}
if ( ndiff ) diff_sites_push(args, ndiff, qry_rec->rid, qry_rec->pos);
}
- else // use_PLs set
+ else // gt_err set
{
for (i=0; i<args->npairs; i++)
{
{
int match = qry_dsg & gt_dsg;
args->hwe_prob[i] += hwe_dsg[match];
+ if ( match ) args->nmatch[i]++;
}
args->ncnt[i]++;
}
}
int idx=0;
- if ( !args->use_PLs )
+ if ( !args->gt_err )
{
for (i=0; i<args->nqry_smpl; i++)
{
if ( !args->gt_dsg[j] ) { idx++; continue; } // missing value
int match = args->qry_dsg[i] & args->gt_dsg[j];
if ( !match ) args->ndiff[idx]++;
- else if ( args->calc_hwe_prob ) args->hwe_prob[idx] += hwe_dsg[match];
+ else if ( args->calc_hwe_prob )
+ {
+ args->hwe_prob[idx] += hwe_dsg[match];
+ args->nmatch[idx]++;
+ }
args->ncnt[idx]++;
idx++;
}
}
}
- else // use_PLs set
+ else // gt_err set
{
for (i=0; i<args->nqry_smpl; i++)
{
{
int match = args->qry_dsg[i] & args->gt_dsg[j];
args->hwe_prob[idx] += hwe_dsg[match];
+ if ( match ) args->nmatch[idx]++;
}
args->ncnt[idx]++;
idx++;
{
extsort_sort(args->es);
- fprintf(args->fp,"# DS, distinctive sites:\n");
- fprintf(args->fp,"# - chromosome\n");
- fprintf(args->fp,"# - position\n");
- fprintf(args->fp,"# - cumulative number of pairs distinguished by this block\n");
- fprintf(args->fp,"# - block id\n");
- fprintf(args->fp,"#DS\t[2]Chromosome\t[3]Position\t[4]Cumulative number of distinct pairs\t[5]Block id\n");
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"# DS, distinctive sites:\n");
+ ksprintf(&args->kstr,"# - chromosome\n");
+ ksprintf(&args->kstr,"# - position\n");
+ ksprintf(&args->kstr,"# - cumulative number of pairs distinguished by this block\n");
+ ksprintf(&args->kstr,"# - block id\n");
+ ksprintf(&args->kstr,"#DS\t[2]Chromosome\t[3]Position\t[4]Cumulative number of distinct pairs\t[5]Block id\n");
kbitset_t *kbs_blk = kbs_init(args->npairs);
kbitset_iter_t itr;
if ( ndiff_dbg!=ndiff ) error("Corrupted data, fixme: %d vs %d\n",ndiff_dbg,ndiff);
if ( !ndiff_new ) continue; // no new pair distinguished by this site
ndiff_tot += ndiff_new;
- fprintf(args->fp,"DS\t%s\t%d\t%d\t%d\n",bcf_hdr_id2name(args->qry_hdr,rid),pos+1,ndiff_tot,iblock);
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"DS\t%s\t%d\t%d\t%d\n",bcf_hdr_id2name(args->qry_hdr,rid),pos+1,ndiff_tot,iblock);
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
if ( ndiff_tot < ndiff_min ) continue; // fewer than the requested number of pairs can be distinguished at this point
iblock++;
ndiff_tot = 0;
}
static void report(args_t *args)
{
- fprintf(args->fp,"INFO\tsites-compared\t%u\n",args->ncmp);
- fprintf(args->fp,"INFO\tsites-skipped-no-match\t%u\n",args->nskip_no_match);
- fprintf(args->fp,"INFO\tsites-skipped-multiallelic\t%u\n",args->nskip_not_ba);
- fprintf(args->fp,"INFO\tsites-skipped-monoallelic\t%u\n",args->nskip_mono);
- fprintf(args->fp,"INFO\tsites-skipped-no-data\t%u\n",args->nskip_no_data);
- fprintf(args->fp,"INFO\tsites-skipped-GT-not-diploid\t%u\n",args->nskip_dip_GT);
- fprintf(args->fp,"INFO\tsites-skipped-PL-not-diploid\t%u\n",args->nskip_dip_PL);
- fprintf(args->fp,"INFO\tsites-used-PL-vs-PL\t%u\n",args->nused[0][0]);
- fprintf(args->fp,"INFO\tsites-used-PL-vs-GT\t%u\n",args->nused[0][1]);
- fprintf(args->fp,"INFO\tsites-used-GT-vs-PL\t%u\n",args->nused[1][0]);
- fprintf(args->fp,"INFO\tsites-used-GT-vs-GT\t%u\n",args->nused[1][1]);
- fprintf(args->fp,"# DC, discordance:\n");
- fprintf(args->fp,"# - query sample\n");
- fprintf(args->fp,"# - genotyped sample\n");
- fprintf(args->fp,"# - discordance (either an abstract score or number of mismatches, see -e/-u in the man page for details; smaller is better)\n");
- fprintf(args->fp,"# - negative log of HWE probability at matching sites (rare genotypes matches are more informative, bigger is better)\n");
- fprintf(args->fp,"# - number of sites compared (bigger is better)\n");
- fprintf(args->fp,"#DC\t[2]Query Sample\t[3]Genotyped Sample\t[4]Discordance\t[5]-log P(HWE)\t[6]Number of sites compared\n");
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"INFO\tsites-compared\t%u\n",args->ncmp);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-no-match\t%u\n",args->nskip_no_match);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-multiallelic\t%u\n",args->nskip_not_ba);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-monoallelic\t%u\n",args->nskip_mono);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-no-data\t%u\n",args->nskip_no_data);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-GT-not-diploid\t%u\n",args->nskip_dip_GT);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-PL-not-diploid\t%u\n",args->nskip_dip_PL);
+ ksprintf(&args->kstr,"INFO\tsites-skipped-filtering-expression\t%u\n",args->nskip_filter);
+ ksprintf(&args->kstr,"INFO\tsites-used-PL-vs-PL\t%u\n",args->nused[0][0]);
+ ksprintf(&args->kstr,"INFO\tsites-used-PL-vs-GT\t%u\n",args->nused[0][1]);
+ ksprintf(&args->kstr,"INFO\tsites-used-GT-vs-PL\t%u\n",args->nused[1][0]);
+ ksprintf(&args->kstr,"INFO\tsites-used-GT-vs-GT\t%u\n",args->nused[1][1]);
+ ksprintf(&args->kstr,"# DCv2, discordance version 2:\n");
+ ksprintf(&args->kstr,"# - Query sample\n");
+ ksprintf(&args->kstr,"# - Genotyped sample\n");
+ ksprintf(&args->kstr,"# - Discordance, given either as an abstract score or number of mismatches, see the options -E/-u\n"
+ "# in man page for details. Note that samples with high missingness have fewer sites compared,\n"
+ "# which results in lower overall discordance. Therefore it is advisable to use the average score\n"
+ "# per site rather than the absolute value, i.e. divide the value by the number of sites compared\n"
+ "# (smaller value = better match)\n");
+ ksprintf(&args->kstr,"# - Average negative log of HWE probability at matching sites, attempts to quantify the following\n"
+ "# intuition: rare genotype matches are more informative than common genotype matches, hence two\n"
+ "# samples with similar discordance can be further stratified by the HWE score (bigger value = better\n"
+ "# match, the observed concordance was less likely to occur by chance)\n");
+ ksprintf(&args->kstr,"# - Number of sites compared for this pair of samples (bigger = more informative)\n");
+ ksprintf(&args->kstr,"# - Number of matching genotypes\n");
+ ksprintf(&args->kstr,"#DCv2\t[2]Query Sample\t[3]Genotyped Sample\t[4]Discordance\t[5]Average -log P(HWE)\t[6]Number of sites compared\t[7]Number of matching genotypes\n");
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
int trim = args->ntop;
if ( !args->pairs )
int i;
for (i=0; i<args->npairs; i++)
{
+ args->kstr.l = 0;
int iqry = args->pairs[i].iqry;
int igt = args->pairs[i].igt;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->ndiff[i],
- args->calc_hwe_prob ? args->hwe_prob[i] : 0,
- args->ncnt[i]);
+ (args->calc_hwe_prob && args->nmatch[i]) ? args->hwe_prob[i]/args->nmatch[i] : 0,
+ args->ncnt[i],
+ args->nmatch[i]);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->pdiff[i],
- args->calc_hwe_prob ? args->hwe_prob[i] : 0,
- args->ncnt[i]);
+ (args->calc_hwe_prob && args->nmatch[i]) ? args->hwe_prob[i]/args->nmatch[i] : 0,
+ args->ncnt[i],
+ args->nmatch[i]);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
}
else if ( !trim )
int ngt = args->cross_check ? i : args->ngt_smpl;
for (j=0; j<ngt; j++)
{
+ args->kstr.l = 0;
int igt = args->gt_smpl ? args->gt_smpl[j] : j;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->ndiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->pdiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
idx++;
}
}
for (j=0; j<args->ngt_smpl; j++)
{
if ( args->sort_by_hwe )
- arr[j].val = -args->hwe_prob[idx];
+ arr[j].val = args->nmatch[idx] ? -args->hwe_prob[idx]/args->nmatch[idx] : 0; // -args->hwe_prob[idx];
else if ( args->ndiff )
arr[j].val = args->ncnt[idx] ? (double)args->ndiff[idx]/args->ncnt[idx] : 0;
else
int iqry = args->qry_smpl ? args->qry_smpl[i] : i;
for (j=0; j<args->ntop; j++)
{
+ args->kstr.l = 0;
int idx = arr[j].idx;
int igt = args->gt_smpl ? args->gt_smpl[arr[j].ism] : arr[j].ism;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->ndiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->gt_hdr?args->gt_hdr->samples[igt]:args->qry_hdr->samples[igt],
args->pdiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
}
free(arr);
for (j=0; j<i; j++)
{
if ( args->sort_by_hwe )
- arr[k].val = -args->hwe_prob[idx];
+ arr[k].val = args->nmatch[idx] ? -args->hwe_prob[idx]/args->nmatch[idx] : 0;
else if ( args->ndiff )
arr[k].val = args->ncnt[idx] ? (double)args->ndiff[idx]/args->ncnt[idx] : 0;
else
{
idx = j*(j+1)/2 + i;
if ( args->sort_by_hwe )
- arr[k].val = -args->hwe_prob[idx];
+ arr[k].val = args->nmatch[idx] ? -args->hwe_prob[idx]/args->nmatch[idx] : 0;
else if ( args->ndiff )
arr[k].val = args->ncnt[idx] ? (double)args->ndiff[idx]/args->ncnt[idx] : 0;
else
int iqry = args->qry_smpl ? args->qry_smpl[i] : i;
for (j=0; j<args->ntop; j++)
{
+ args->kstr.l = 0;
if ( i <= arr[j].ism ) continue;
int idx = arr[j].idx;
int igt = args->qry_smpl ? args->qry_smpl[arr[j].ism] : arr[j].ism;
if ( args->ndiff )
{
- fprintf(args->fp,"DC\t%s\t%s\t%u\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%u\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->qry_hdr->samples[igt],
args->ndiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
else
{
- fprintf(args->fp,"DC\t%s\t%s\t%e\t%e\t%u\n",
+ ksprintf(&args->kstr,"DCv2\t%s\t%s\t%e\t%e\t%u\t%u\n",
args->qry_hdr->samples[iqry],
args->qry_hdr->samples[igt],
args->pdiff[idx],
- args->calc_hwe_prob ? args->hwe_prob[idx] : 0,
- args->ncnt[idx]);
+ (args->calc_hwe_prob && args->nmatch[idx]) ? args->hwe_prob[idx]/args->nmatch[idx] : 0,
+ args->ncnt[idx],
+ args->calc_hwe_prob ? args->nmatch[idx] : 0);
}
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
}
}
free(arr);
fprintf(bcftools_stderr, " and TMP is a prefix of temporary files used by external sorting [/tmp/bcftools.XXXXXX]\n");
#endif
fprintf(bcftools_stderr, " --dry-run Stop after first record to estimate required time\n");
- fprintf(bcftools_stderr, " -e, --error-probability INT Phred-scaled probability of genotyping error, 0 for faster but less accurate results [40]\n");
+ fprintf(bcftools_stderr, " -E, --error-probability INT Phred-scaled probability of genotyping error, 0 for faster but less accurate results [40]\n");
+ fprintf(bcftools_stderr, " -e, --exclude [qry|gt]:EXPR Exclude sites for which the expression is true\n");
fprintf(bcftools_stderr, " -g, --genotypes FILE Genotypes to compare against\n");
fprintf(bcftools_stderr, " -H, --homs-only Homozygous genotypes only, useful with low coverage data (requires -g)\n");
+ fprintf(bcftools_stderr, " -i, --include [qry|gt]:EXPR Include sites for which the expression is true\n");
fprintf(bcftools_stderr, " --n-matches INT Print only top INT matches for each sample (sorted by average score), 0 for unlimited.\n");
fprintf(bcftools_stderr, " Use negative value to sort by HWE probability rather than by discordance [0]\n");
fprintf(bcftools_stderr, " --no-HWE-prob Disable calculation of HWE probability\n");
+ fprintf(bcftools_stderr, " -o, --output FILE Write output to a file [standard output]\n");
+ fprintf(bcftools_stderr, " -O, --output-type t|z t: plain tab-delimited text output, z: compressed [t]\n");
fprintf(bcftools_stderr, " -p, --pairs LIST Comma-separated sample pairs to compare (qry,gt[,qry,gt..] with -g or qry,qry[,qry,qry..] w/o)\n");
fprintf(bcftools_stderr, " -P, --pairs-file FILE File with tab-delimited sample pairs to compare (qry,gt with -g or qry,qry w/o)\n");
fprintf(bcftools_stderr, " -r, --regions REGION Restrict to comma-separated list of regions\n");
fprintf(bcftools_stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
fprintf(bcftools_stderr, " -u, --use TAG1[,TAG2] Which tag to use in the query file (TAG1) and the -g file (TAG2) [PL,GT]\n");
fprintf(bcftools_stderr, "Examples:\n");
- fprintf(bcftools_stderr, " # Check discordance of all samples from B against all sample in A\n");
+ fprintf(bcftools_stderr, " # Check discordance of all samples from B against all samples in A\n");
fprintf(bcftools_stderr, " bcftools gtcheck -g A.bcf B.bcf\n");
fprintf(bcftools_stderr, "\n");
- fprintf(bcftools_stderr, " # Limit comparisons to the fiven list of samples\n");
+ fprintf(bcftools_stderr, " # Limit comparisons to the given list of samples\n");
fprintf(bcftools_stderr, " bcftools gtcheck -s gt:a1,a2,a3 -s qry:b1,b2 -g A.bcf B.bcf\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, " # Compare only two pairs a1,b1 and a1,b2\n");
args->qry_use_GT = -1;
args->gt_use_GT = -1;
args->calc_hwe_prob = 1;
- args->use_PLs = 40;
+ args->gt_err = 40;
args->regions_overlap = 1;
args->targets_overlap = 0;
+ args->output_fname = "-";
// external sort for --distinctive-sites
#ifdef _WIN32
static struct option loptions[] =
{
- {"error-probability",1,0,'e'},
+ {"error-probability",1,0,'E'}, // note this used to be 'e', but can easily auto-detect to assure backward compatibility
+ {"exclude",required_argument,0,'e'},
+ {"include",required_argument,0,'i'},
+ {"output",required_argument,0,'o'},
+ {"output-type",required_argument,NULL,'O'},
{"use",1,0,'u'},
{"cluster",1,0,'c'},
{"GTs-only",1,0,'G'},
{0,0,0,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hg:p:s:S:p:P:Hr:R:at:T:G:c:u:e:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hg:p:s:S:p:P:Hr:R:at:T:G:c:u:e:E:i:o:O:",loptions,NULL)) >= 0) {
switch (c) {
+ case 'o': args->output_fname = optarg; break;
+ case 'O':
+ switch (optarg[0]) {
+ case 't': args->output_type = FT_TAB_TEXT; break;
+ case 'z': args->output_type = FT_VCF_GZ; break;
+ default:
+ {
+ args->clevel = strtol(optarg,&tmp,10);
+ if ( *tmp || args->clevel<0 || args->clevel>9 ) error("The output type \"%s\" not recognised\n", optarg);
+ }
+ }
+ if ( optarg[1] )
+ {
+ args->clevel = strtol(optarg+1,&tmp,10);
+ if ( *tmp || args->clevel<0 || args->clevel>9 ) error("Could not parse argument: --output-type %s\n", optarg+1);
+ }
+ break;
case 'e':
- args->use_PLs = strtol(optarg,&tmp,10);
+ if ( !strncasecmp("gt:",optarg,3) )
+ {
+ if ( args->gt_filter_str ) error("Error: only one -i or -e expression can be given for gt:, and they cannot be combined\n");
+ args->gt_filter_str = optarg;
+ args->gt_filter_logic |= FLT_EXCLUDE;
+ }
+ else if ( !strncasecmp("qry:",optarg,4) )
+ {
+ if ( args->qry_filter_str ) error("Error: only one -i or -e expression can be given for qry:, and they cannot be combined\n");
+ args->qry_filter_str = optarg;
+ args->qry_filter_logic |= FLT_EXCLUDE;
+ }
+ else
+ {
+ // this could be the old -e, --error-probability option
+ args->gt_err = strtol(optarg,&tmp,10);
+ if ( !tmp || *tmp )
+ {
+ // it is not
+ args->gt_filter_str = optarg;
+ args->qry_filter_str = optarg;
+ args->gt_filter_logic |= FLT_EXCLUDE;
+ args->qry_filter_logic |= FLT_EXCLUDE;
+ }
+ else
+ {
+ fprintf(bcftools_stderr,"[warning] auto-detected the old format --error-probability option, please switch from -e to -E.\n");
+ }
+ }
+ break;
+ case 'i':
+ if ( !strncasecmp("gt:",optarg,3) )
+ {
+ if ( args->gt_filter_str ) error("Error: only one -i or -e expression can be given for gt:, and they cannot be combined\n");
+ args->gt_filter_str = optarg;
+ args->gt_filter_logic |= FLT_INCLUDE;
+ }
+ else if ( !strncasecmp("qry:",optarg,4) )
+ {
+ if ( args->qry_filter_str ) error("Error: only one -i or -e expression can be given for qry:, and they cannot be combined\n");
+ args->qry_filter_str = optarg;
+ args->qry_filter_logic |= FLT_INCLUDE;
+ }
+ else
+ {
+ args->gt_filter_str = optarg;
+ args->qry_filter_str = optarg;
+ args->gt_filter_logic |= FLT_INCLUDE;
+ args->qry_filter_logic |= FLT_INCLUDE;
+ }
+ break;
+ case 'E':
+ args->gt_err = strtol(optarg,&tmp,10);
if ( !tmp || *tmp ) error("Could not parse: --error-probability %s\n", optarg);
break;
case 'u':
while ( *tmp && *tmp!=',' ) tmp++;
if ( *tmp ) { *tmp = 0; args->es_tmp_prefix = tmp+1; }
}
- args->use_PLs = 0;
+ args->gt_err = 0;
break;
case 'c':
error("The -c option is to be implemented, please open an issue on github\n");
}
if ( args->distinctive_sites && !args->pair_samples ) error("The experimental option --distinctive-sites requires -p/-P\n");
if ( args->hom_only && !args->gt_fname ) error("The option --homs-only requires --genotypes\n");
- if ( args->distinctive_sites && args->use_PLs ) error("The option --distinctive-sites cannot be combined with --error-probability\n");
+ if ( args->distinctive_sites && args->gt_err ) error("The option --distinctive-sites cannot be combined with --error-probability\n");
init_data(args);
gettimeofday(&t1, NULL);
double delta = (t1.tv_sec - t0.tv_sec) * 1e6 + (t1.tv_usec - t0.tv_usec);
fprintf(bcftools_stderr,"INFO:\tTime required to process one record .. %f seconds\n",delta/1e6);
- fprintf(args->fp,"INFO\tTime required to process one record .. %f seconds\n",delta/1e6);
+ args->kstr.l = 0;
+ ksprintf(&args->kstr,"INFO\tTime required to process one record .. %f seconds\n",delta/1e6);
+ if ( bgzf_write(args->out_fh, args->kstr.s, args->kstr.l)!=args->kstr.l ) error("Failed to write to %s\n", args->output_fname);
if ( args->dry_run ) break;
}
}
/* vcfhead.c -- view VCF/BCF file headers.
Copyright (C) 2021 University of Glasgow.
+ Copyright (C) 2023 Genome Research Ltd.
Author: John Marshall <jmarshall@hey.com>
"Usage: bcftools head [OPTION]... [FILE]\n"
"\n"
"Options:\n"
-" -h, --headers INT Display INT header lines [all]\n"
-" -n, --records INT Display INT variant record lines [none]\n"
+" -h, --headers INT Display INT header lines [all]\n"
+" -n, --records INT Display INT variant record lines [none]\n"
+" -s, --samples INT Display INT records starting with the #CHROM header line [none]\n"
"\n";
static const struct option loptions[] = {
{ "headers", required_argument, NULL, 'h' },
{ "records", required_argument, NULL, 'n' },
+ { "samples", required_argument, NULL, 's' },
{ NULL, 0, NULL, 0 }
};
int all_headers = 1;
+ int samples = 0;
uint64_t nheaders = 0;
uint64_t nrecords = 0;
int c, nargs;
- while ((c = getopt_long(argc, argv, "h:n:", loptions, NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h:n:s:", loptions, NULL)) >= 0)
switch (c) {
case 'h': all_headers = 0; nheaders = strtoull(optarg, NULL, 0); break;
case 'n': nrecords = strtoull(optarg, NULL, 0); break;
+ case 's': nrecords = strtoull(optarg, NULL, 0); samples = 1; break;
default:
fputs(usage, stderr);
return EXIT_FAILURE;
}
+ if ( samples && all_headers ) all_headers = 0;
+
nargs = argc - optind;
if (nargs == 0 && isatty(STDIN_FILENO)) {
fputs(usage, stdout);
bcf_hdr_format(hdr, 0, &str);
fputs(ks_str(&str), stdout);
}
- else if (nheaders > 0) {
+ else if (nheaders > 0 || samples ) {
bcf_hdr_format(hdr, 0, &str);
char *lim = str.s;
uint64_t n;
+ int samples_printed = 0;
for (n = 0; n < nheaders; n++) {
+ if ( samples && !strncmp(lim,"#CHROM\t",7) ) samples_printed = 1;
lim = strchr(lim, '\n');
if (lim) lim++;
else break;
}
- if (lim) *lim = '\0';
- fputs(ks_str(&str), stdout);
+ if ( nheaders )
+ {
+ char tmp;
+ if (lim) { tmp = *lim; *lim = '\0'; }
+ fputs(ks_str(&str), stdout);
+ if (lim) *lim = tmp;
+ }
+ if ( lim && samples && !samples_printed )
+ {
+ while ( lim && *lim )
+ {
+ if ( !strncmp(lim,"#CHROM\t",7) ) { fputs(lim, stdout); break; }
+ lim = strchr(lim, '\n');
+ if (lim) lim++;
+ else break;
+ }
+ }
}
if (nrecords > 0) {
/* vcfhead.c -- view VCF/BCF file headers.
Copyright (C) 2021 University of Glasgow.
+ Copyright (C) 2023 Genome Research Ltd.
Author: John Marshall <jmarshall@hey.com>
"Usage: bcftools head [OPTION]... [FILE]\n"
"\n"
"Options:\n"
-" -h, --headers INT Display INT header lines [all]\n"
-" -n, --records INT Display INT variant record lines [none]\n"
+" -h, --headers INT Display INT header lines [all]\n"
+" -n, --records INT Display INT variant record lines [none]\n"
+" -s, --samples INT Display INT records starting with the #CHROM header line [none]\n"
"\n";
static const struct option loptions[] = {
{ "headers", required_argument, NULL, 'h' },
{ "records", required_argument, NULL, 'n' },
+ { "samples", required_argument, NULL, 's' },
{ NULL, 0, NULL, 0 }
};
int all_headers = 1;
+ int samples = 0;
uint64_t nheaders = 0;
uint64_t nrecords = 0;
int c, nargs;
- while ((c = getopt_long(argc, argv, "h:n:", loptions, NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h:n:s:", loptions, NULL)) >= 0)
switch (c) {
case 'h': all_headers = 0; nheaders = strtoull(optarg, NULL, 0); break;
case 'n': nrecords = strtoull(optarg, NULL, 0); break;
+ case 's': nrecords = strtoull(optarg, NULL, 0); samples = 1; break;
default:
fputs(usage, bcftools_stderr);
return EXIT_FAILURE;
}
+ if ( samples && all_headers ) all_headers = 0;
+
nargs = argc - optind;
if (nargs == 0 && isatty(STDIN_FILENO)) {
fputs(usage, bcftools_stdout);
bcf_hdr_format(hdr, 0, &str);
fputs(ks_str(&str), bcftools_stdout);
}
- else if (nheaders > 0) {
+ else if (nheaders > 0 || samples ) {
bcf_hdr_format(hdr, 0, &str);
char *lim = str.s;
uint64_t n;
+ int samples_printed = 0;
for (n = 0; n < nheaders; n++) {
+ if ( samples && !strncmp(lim,"#CHROM\t",7) ) samples_printed = 1;
lim = strchr(lim, '\n');
if (lim) lim++;
else break;
}
- if (lim) *lim = '\0';
- fputs(ks_str(&str), bcftools_stdout);
+ if ( nheaders )
+ {
+ char tmp;
+ if (lim) { tmp = *lim; *lim = '\0'; }
+ fputs(ks_str(&str), bcftools_stdout);
+ if (lim) *lim = tmp;
+ }
+ if ( lim && samples && !samples_printed )
+ {
+ while ( lim && *lim )
+ {
+ if ( !strncmp(lim,"#CHROM\t",7) ) { fputs(lim, bcftools_stdout); break; }
+ lim = strchr(lim, '\n');
+ if (lim) lim++;
+ else break;
+ }
+ }
}
if (nrecords > 0) {
/* vcfindex.c -- Index bgzip compressed VCF/BCF files for random access.
- Copyright (C) 2014-2021 Genome Research Ltd.
+ Copyright (C) 2014-2024 Genome Research Ltd.
Author: Shane McCarthy <sm15@sanger.ac.uk>
default: usage();
}
}
+ if (!min_shift) tbi = 1;
if (stats > total)
{
fprintf(stderr, "[E::%s] expected only one of --stats or --nrecords options\n", __func__);
/* vcfindex.c -- Index bgzip compressed VCF/BCF files for random access.
- Copyright (C) 2014-2021 Genome Research Ltd.
+ Copyright (C) 2014-2024 Genome Research Ltd.
Author: Shane McCarthy <sm15@sanger.ac.uk>
default: usage();
}
}
+ if (!min_shift) tbi = 1;
if (stats > total)
{
fprintf(bcftools_stderr, "[E::%s] expected only one of --stats or --nrecords options\n", __func__);
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/hts_os.h>
+#include <htslib/hts_defs.h>
#include "bcftools.h"
#include "filter.h"
FILE *fh_log, *fh_sites;
htsFile **fh_out;
char **argv, *prefix, *output_fname, **fnames, *write_files, *targets_list, *regions_list;
- char *isec_exact;
+ char *isec_exact, *file_list;
int argc, record_cmd_line;
char *index_fn;
int write_index;
* mkdir_p() - create new directory for a file $fname
* @fname: the file name to create the directory for, the part after last "/" is ignored
*/
-void mkdir_p(const char *fmt, ...)
+void HTS_FORMAT(HTS_PRINTF_FMT, 1, 2)
+mkdir_p(const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
int n = vsnprintf(NULL, 0, fmt, ap) + 2;
va_end(ap);
- char *path = (char*)malloc(n);
+ char *tmp = (char*)malloc(n);
+ if (!tmp) error("Couldn't allocate space for path: %s\n", strerror(errno));
va_start(ap, fmt);
- vsnprintf(path, n, fmt, ap);
+ vsnprintf(tmp, n, fmt, ap);
va_end(ap);
- char *tmp = strdup(path), *p = tmp+1;
+ char *p = tmp+1;
while (*p)
{
while (*p && *p!='/') p++;
char ctmp = *p;
*p = 0;
int ret = mkdir(tmp,S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH);
- if ( ret!=0 && errno!=EEXIST ) error("Error creating directory %s: %s\n", path,strerror(errno));
+ if ( ret!=0 && errno!=EEXIST ) error("Error creating directory %s: %s\n", tmp,strerror(errno));
*p = ctmp;
while ( *p && *p=='/' ) p++;
}
free(tmp);
- free(path);
}
/**
*
* Returns open file descriptor or NULL if mode is NULL.
*/
-FILE *open_file(char **fname, const char *mode, const char *fmt, ...)
+FILE * HTS_FORMAT(HTS_PRINTF_FMT, 3, 4)
+open_file(char **fname, const char *mode, const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
vsnprintf(str, n, fmt, ap);
va_end(ap);
- mkdir_p(str);
+ mkdir_p("%s", str);
if ( !mode )
{
if ( !fname ) error("Uh: expected fname or mode\n");
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if (args->record_cmd_line) bcf_hdr_append_version(files->readers[args->iwrite].header,args->argc,args->argv,"bcftools_isec");
if ( bcf_hdr_write(out_fh, files->readers[args->iwrite].header)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname?args->output_fname:"standard output");
- if ( args->write_index && init_index(out_fh,files->readers[args->iwrite].header,args->output_fname,&args->index_fn)<0 )
- error("Error: failed to initialise index for %s\n",args->output_fname?args->output_fname:"standard output");
+ if ( init_index2(out_fh,files->readers[args->iwrite].header,
+ args->output_fname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",
+ args->output_fname?args->output_fname:"standard output");
}
if ( !args->nwrite && !out_std && !args->prefix )
fprintf(stderr,"Note: -w option not given, printing list of sites...\n");
{
if ( !args->fnames[i] ) continue;
if ( hts_close(args->fh_out[i])!=0 ) error("[%s] Error: close failed .. %s\n", __func__,args->fnames[i]);
- if ( args->output_type==FT_VCF_GZ )
+ int is_tbi = !args->write_index
+ || (args->write_index&127) == HTS_FMT_TBI;
+ if ( args->output_type==FT_VCF_GZ && is_tbi )
{
tbx_conf_t conf = tbx_conf_vcf;
tbx_index_build(args->fnames[i], -1, &conf);
}
- else if ( args->output_type==FT_BCF_GZ )
+ else if ( args->output_type==FT_BCF_GZ || !is_tbi )
{
if ( bcf_index_build(args->fnames[i],14) ) error("Could not index %s\n", args->fnames[i]);
}
fprintf(stderr, " -e, --exclude EXPR Exclude sites for which the expression is true\n");
fprintf(stderr, " -f, --apply-filters LIST Require at least one of the listed FILTER strings (e.g. \"PASS,.\")\n");
fprintf(stderr, " -i, --include EXPR Include only sites for which the expression is true\n");
+ fprintf(stderr, " -l, --file-list FILE Read the input file names from the file\n");
fprintf(stderr, " --no-version Do not append version and command line to the header\n");
fprintf(stderr, " -n, --nfiles [+-=~]INT Output positions present in this many (=), this many or more (+), this many or fewer (-), the exact (~) files\n");
fprintf(stderr, " -o, --output FILE Write output to a file [standard output]\n");
fprintf(stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
fprintf(stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
fprintf(stderr, " -w, --write LIST List of files to write with -p given as 1-based indexes. By default, all files are written\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Examples:\n");
fprintf(stderr, " # Create intersection and complements of two sets saving the output in dir/*\n");
{"collapse",required_argument,NULL,'c'},
{"complement",no_argument,NULL,'C'},
{"apply-filters",required_argument,NULL,'f'},
+ {"file-list",required_argument,NULL,'l'},
{"nfiles",required_argument,NULL,'n'},
{"prefix",required_argument,NULL,'p'},
{"write",required_argument,NULL,'w'},
{"output-type",required_argument,NULL,'O'},
{"threads",required_argument,NULL,9},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hc:r:R:p:n:w:t:T:Cf:o:O:i:e:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hc:r:R:p:n:w:t:T:Cf:o:O:i:e:l:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'o': args->output_fname = optarg; break;
case 'O':
case 'C':
if ( args->isec_op!=0 && args->isec_op!=OP_COMPLEMENT ) error("Error: either -C or -n should be given, not both.\n");
args->isec_op = OP_COMPLEMENT; break;
+ case 'l': args->file_list = optarg; break;
case 'r': args->regions_list = optarg; break;
case 'R': args->regions_list = optarg; regions_is_file = 1; break;
case 't': args->targets_list = optarg; break;
case 'T': args->targets_list = optarg; targets_is_file = 1; break;
case 'p': args->prefix = optarg; break;
- case 'w': args->write_files = optarg; break;
+ case 'w':
+ if ( args->write_files ) error("The option -w accepts a list of indices and can be given only once\n");
+ args->write_files = optarg;
+ break;
case 'i': add_filter(args, optarg, FLT_INCLUDE); break;
case 'e': add_filter(args, optarg, FLT_EXCLUDE); break;
case 'n':
break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
- case 10 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(); break;
default: error("Unknown argument: %s\n", optarg);
}
}
- if ( argc-optind<1 ) usage(); // no file given
+ if ( argc-optind<1 && !args->file_list ) usage(); // no file given
+
+ int nfiles = 0,i;
+ char **files = NULL;
+ if ( args->file_list )
+ {
+ files = hts_readlines(args->file_list, &nfiles);
+ if ( !files ) error("Failed to read from %s\n", args->file_list);
+ }
+ if ( optind<argc )
+ {
+ int n = argc - optind;
+ files = (char**)realloc(files,sizeof(*files)*(n+nfiles));
+ for (i=nfiles; i>0; i--) files[n+i-1] = files[n+i-2];
+ for (i=0; i<n; i++) files[i] = strdup(argv[optind+i]);
+ nfiles += n;
+ }
+
if ( args->targets_list )
{
bcf_sr_set_opt(args->files,BCF_SR_TARGETS_OVERLAP,targets_overlap);
if ( bcf_sr_set_regions(args->files, args->regions_list, regions_is_file)<0 )
error("Failed to read the regions: %s\n", args->regions_list);
}
- if ( argc-optind==2 && !args->isec_op )
+ if ( nfiles==2 && !args->isec_op )
{
args->isec_op = OP_VENN;
if ( !args->prefix ) error("Expected the -p option\n");
args->isec_n = 1;
}
args->files->require_index = 1;
- while (optind<argc)
+ for (i=0; i<nfiles; i++)
{
- if ( !bcf_sr_add_reader(args->files, argv[optind]) ) error("Failed to open %s: %s\n", argv[optind],bcf_sr_strerror(args->files->errnum));
- optind++;
+ if ( !bcf_sr_add_reader(args->files, files[i]) ) error("Failed to open %s: %s\n", files[i],bcf_sr_strerror(args->files->errnum));
+ free(files[i]);
}
+ free(files);
+
init_data(args);
isec_vcf(args);
destroy_data(args);
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/hts_os.h>
+#include <htslib/hts_defs.h>
#include "bcftools.h"
#include "filter.h"
FILE *fh_log, *fh_sites;
htsFile **fh_out;
char **argv, *prefix, *output_fname, **fnames, *write_files, *targets_list, *regions_list;
- char *isec_exact;
+ char *isec_exact, *file_list;
int argc, record_cmd_line;
char *index_fn;
int write_index;
* mkdir_p() - create new directory for a file $fname
* @fname: the file name to create the directory for, the part after last "/" is ignored
*/
-void mkdir_p(const char *fmt, ...)
+void HTS_FORMAT(HTS_PRINTF_FMT, 1, 2)
+mkdir_p(const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
int n = vsnprintf(NULL, 0, fmt, ap) + 2;
va_end(ap);
- char *path = (char*)malloc(n);
+ char *tmp = (char*)malloc(n);
+ if (!tmp) error("Couldn't allocate space for path: %s\n", strerror(errno));
va_start(ap, fmt);
- vsnprintf(path, n, fmt, ap);
+ vsnprintf(tmp, n, fmt, ap);
va_end(ap);
- char *tmp = strdup(path), *p = tmp+1;
+ char *p = tmp+1;
while (*p)
{
while (*p && *p!='/') p++;
char ctmp = *p;
*p = 0;
int ret = mkdir(tmp,S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH);
- if ( ret!=0 && errno!=EEXIST ) error("Error creating directory %s: %s\n", path,strerror(errno));
+ if ( ret!=0 && errno!=EEXIST ) error("Error creating directory %s: %s\n", tmp,strerror(errno));
*p = ctmp;
while ( *p && *p=='/' ) p++;
}
free(tmp);
- free(path);
}
/**
*
* Returns open file descriptor or NULL if mode is NULL.
*/
-FILE *open_file(char **fname, const char *mode, const char *fmt, ...)
+FILE * HTS_FORMAT(HTS_PRINTF_FMT, 3, 4)
+open_file(char **fname, const char *mode, const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
vsnprintf(str, n, fmt, ap);
va_end(ap);
- mkdir_p(str);
+ mkdir_p("%s", str);
if ( !mode )
{
if ( !fname ) error("Uh: expected fname or mode\n");
if ( args->n_threads ) hts_set_threads(out_fh, args->n_threads);
if (args->record_cmd_line) bcf_hdr_append_version(files->readers[args->iwrite].header,args->argc,args->argv,"bcftools_isec");
if ( bcf_hdr_write(out_fh, files->readers[args->iwrite].header)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname?args->output_fname:"standard output");
- if ( args->write_index && init_index(out_fh,files->readers[args->iwrite].header,args->output_fname,&args->index_fn)<0 )
- error("Error: failed to initialise index for %s\n",args->output_fname?args->output_fname:"standard output");
+ if ( init_index2(out_fh,files->readers[args->iwrite].header,
+ args->output_fname,&args->index_fn,
+ args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",
+ args->output_fname?args->output_fname:"standard output");
}
if ( !args->nwrite && !out_std && !args->prefix )
fprintf(bcftools_stderr,"Note: -w option not given, printing list of sites...\n");
{
if ( !args->fnames[i] ) continue;
if ( hts_close(args->fh_out[i])!=0 ) error("[%s] Error: close failed .. %s\n", __func__,args->fnames[i]);
- if ( args->output_type==FT_VCF_GZ )
+ int is_tbi = !args->write_index
+ || (args->write_index&127) == HTS_FMT_TBI;
+ if ( args->output_type==FT_VCF_GZ && is_tbi )
{
tbx_conf_t conf = tbx_conf_vcf;
tbx_index_build(args->fnames[i], -1, &conf);
}
- else if ( args->output_type==FT_BCF_GZ )
+ else if ( args->output_type==FT_BCF_GZ || !is_tbi )
{
if ( bcf_index_build(args->fnames[i],14) ) error("Could not index %s\n", args->fnames[i]);
}
fprintf(bcftools_stderr, " -e, --exclude EXPR Exclude sites for which the expression is true\n");
fprintf(bcftools_stderr, " -f, --apply-filters LIST Require at least one of the listed FILTER strings (e.g. \"PASS,.\")\n");
fprintf(bcftools_stderr, " -i, --include EXPR Include only sites for which the expression is true\n");
+ fprintf(bcftools_stderr, " -l, --file-list FILE Read the input file names from the file\n");
fprintf(bcftools_stderr, " --no-version Do not append version and command line to the header\n");
fprintf(bcftools_stderr, " -n, --nfiles [+-=~]INT Output positions present in this many (=), this many or more (+), this many or fewer (-), the exact (~) files\n");
fprintf(bcftools_stderr, " -o, --output FILE Write output to a file [standard output]\n");
fprintf(bcftools_stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
fprintf(bcftools_stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
fprintf(bcftools_stderr, " -w, --write LIST List of files to write with -p given as 1-based indexes. By default, all files are written\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Examples:\n");
fprintf(bcftools_stderr, " # Create intersection and complements of two sets saving the output in dir/*\n");
{"collapse",required_argument,NULL,'c'},
{"complement",no_argument,NULL,'C'},
{"apply-filters",required_argument,NULL,'f'},
+ {"file-list",required_argument,NULL,'l'},
{"nfiles",required_argument,NULL,'n'},
{"prefix",required_argument,NULL,'p'},
{"write",required_argument,NULL,'w'},
{"output-type",required_argument,NULL,'O'},
{"threads",required_argument,NULL,9},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hc:r:R:p:n:w:t:T:Cf:o:O:i:e:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hc:r:R:p:n:w:t:T:Cf:o:O:i:e:l:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'o': args->output_fname = optarg; break;
case 'O':
case 'C':
if ( args->isec_op!=0 && args->isec_op!=OP_COMPLEMENT ) error("Error: either -C or -n should be given, not both.\n");
args->isec_op = OP_COMPLEMENT; break;
+ case 'l': args->file_list = optarg; break;
case 'r': args->regions_list = optarg; break;
case 'R': args->regions_list = optarg; regions_is_file = 1; break;
case 't': args->targets_list = optarg; break;
case 'T': args->targets_list = optarg; targets_is_file = 1; break;
case 'p': args->prefix = optarg; break;
- case 'w': args->write_files = optarg; break;
+ case 'w':
+ if ( args->write_files ) error("The option -w accepts a list of indices and can be given only once\n");
+ args->write_files = optarg;
+ break;
case 'i': add_filter(args, optarg, FLT_INCLUDE); break;
case 'e': add_filter(args, optarg, FLT_EXCLUDE); break;
case 'n':
break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
- case 10 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(); break;
default: error("Unknown argument: %s\n", optarg);
}
}
- if ( argc-optind<1 ) usage(); // no file given
+ if ( argc-optind<1 && !args->file_list ) usage(); // no file given
+
+ int nfiles = 0,i;
+ char **files = NULL;
+ if ( args->file_list )
+ {
+ files = hts_readlines(args->file_list, &nfiles);
+ if ( !files ) error("Failed to read from %s\n", args->file_list);
+ }
+ if ( optind<argc )
+ {
+ int n = argc - optind;
+ files = (char**)realloc(files,sizeof(*files)*(n+nfiles));
+ for (i=nfiles; i>0; i--) files[n+i-1] = files[n+i-2];
+ for (i=0; i<n; i++) files[i] = strdup(argv[optind+i]);
+ nfiles += n;
+ }
+
if ( args->targets_list )
{
bcf_sr_set_opt(args->files,BCF_SR_TARGETS_OVERLAP,targets_overlap);
if ( bcf_sr_set_regions(args->files, args->regions_list, regions_is_file)<0 )
error("Failed to read the regions: %s\n", args->regions_list);
}
- if ( argc-optind==2 && !args->isec_op )
+ if ( nfiles==2 && !args->isec_op )
{
args->isec_op = OP_VENN;
if ( !args->prefix ) error("Expected the -p option\n");
args->isec_n = 1;
}
args->files->require_index = 1;
- while (optind<argc)
+ for (i=0; i<nfiles; i++)
{
- if ( !bcf_sr_add_reader(args->files, argv[optind]) ) error("Failed to open %s: %s\n", argv[optind],bcf_sr_strerror(args->files->errnum));
- optind++;
+ if ( !bcf_sr_add_reader(args->files, files[i]) ) error("Failed to open %s: %s\n", files[i],bcf_sr_strerror(args->files->errnum));
+ free(files[i]);
}
+ free(files);
+
init_data(args);
isec_vcf(args);
destroy_data(args);
/* vcfmerge.c -- Merge multiple VCF/BCF files to create one multi-sample file.
- Copyright (C) 2012-2023 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/faidx.h>
+#include <htslib/kbitset.h>
+#include <htslib/hts_endian.h>
#include <math.h>
#include <ctype.h>
#include <time.h>
maux_t *maux;
regidx_t *regs; // apply regions only after the blocks are expanded
regitr_t *regs_itr;
- int header_only, collapse, output_type, force_samples, merge_by_id, do_gvcf, filter_logic, missing_to_ref, no_index;
+ int header_only, collapse, output_type, force_samples, force_single, merge_by_id, do_gvcf, filter_logic, missing_to_ref, no_index;
char *header_fname, *output_fname, *regions_list, *info_rules, *file_list;
faidx_t *gvcf_fai;
info_rule_t *rules;
int keep_AC_AN;
char *index_fn;
int write_index;
+ int trim_star_allele; // 0=don't trim; 1=trim at variant sites; 2=trim at all sites
}
args_t;
if ( str.l ) kputc(',',&str);
kputs("QS:sum",&str);
}
+ if ( args->do_gvcf && bcf_hdr_idinfo_exists(args->out_hdr,BCF_HL_INFO,bcf_hdr_id2int(args->out_hdr, BCF_DT_ID, "MIN_DP")) )
+ {
+ if ( str.l ) kputc(',',&str);
+ kputs("MIN_DP:min",&str);
+ }
if ( args->do_gvcf && bcf_hdr_idinfo_exists(args->out_hdr,BCF_HL_INFO,bcf_hdr_id2int(args->out_hdr, BCF_DT_ID, "MinDP")) )
{
if ( str.l ) kputc(',',&str);
if ( len==BCF_VL_A || len==BCF_VL_R )
{
int ifrom = len==BCF_VL_A ? 1 : 0;
- #define BRANCH(type_t, is_missing, is_vector_end, out_type_t) { \
- type_t *src = (type_t *) info->vptr; \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, out_type_t) { \
+ uint8_t *src = info->vptr; \
out_type_t *tgt = (out_type_t *) agr->buf; \
int iori, inew; \
- for (iori=ifrom; iori<line->n_allele; iori++) \
+ for (iori=ifrom; iori<line->n_allele; iori++, src += sizeof(type_t)) \
{ \
+ type_t val = convert(src); \
if ( is_vector_end ) break; \
if ( is_missing ) continue; \
inew = als->map[iori] - ifrom; \
- tgt[inew] = *src; \
- src++; \
+ tgt[inew] = val; \
} \
}
switch (info->type) {
- case BCF_BT_INT8: BRANCH(int8_t, *src==bcf_int8_missing, *src==bcf_int8_vector_end, int); break;
- case BCF_BT_INT16: BRANCH(int16_t, *src==bcf_int16_missing, *src==bcf_int16_vector_end, int); break;
- case BCF_BT_INT32: BRANCH(int32_t, *src==bcf_int32_missing, *src==bcf_int32_vector_end, int); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(*src), bcf_float_is_vector_end(*src), float); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, int); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, int); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, int); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), float); break;
default: fprintf(stderr,"TODO: %s:%d .. info->type=%d\n", __FILE__,__LINE__, info->type); exit(1);
}
#undef BRANCH
}
else
{
- #define BRANCH(type_t, is_missing, is_vector_end, out_type_t) { \
- type_t *src = (type_t *) info->vptr; \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, out_type_t) { \
+ uint8_t *src = info->vptr; \
out_type_t *tgt = (out_type_t *) agr->buf; \
int iori,jori, inew,jnew; \
for (iori=0; iori<line->n_allele; iori++) \
{ \
jnew = als->map[jori]; \
int kori = iori*(iori+1)/2 + jori; \
+ type_t val = convert(&src[kori * sizeof(type_t)]); \
if ( is_vector_end ) break; \
if ( is_missing ) continue; \
int knew = inew>jnew ? inew*(inew+1)/2 + jnew : jnew*(jnew+1)/2 + inew; \
- tgt[knew] = src[kori]; \
+ tgt[knew] = val; \
} \
if ( jori<=iori ) break; \
} \
}
switch (info->type) {
- case BCF_BT_INT8: BRANCH(int8_t, src[kori]==bcf_int8_missing, src[kori]==bcf_int8_vector_end, int); break;
- case BCF_BT_INT16: BRANCH(int16_t, src[kori]==bcf_int16_missing, src[kori]==bcf_int16_vector_end, int); break;
- case BCF_BT_INT32: BRANCH(int32_t, src[kori]==bcf_int32_missing, src[kori]==bcf_int32_vector_end, int); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(src[kori]), bcf_float_is_vector_end(src[kori]), float); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, int); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, int); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, int); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), float); break;
default: fprintf(stderr,"TODO: %s:%d .. info->type=%d\n", __FILE__,__LINE__, info->type); exit(1);
}
#undef BRANCH
{
int i,j, max_ploidy = 0;
- #define BRANCH(type_t, vector_end) { \
- type_t *ptr = (type_t*) fmt->p; \
+ #define BRANCH(type_t, convert, vector_end) { \
+ uint8_t *ptr = fmt->p; \
for (i=0; i<nsmpl; i++) \
{ \
for (j=0; j<fmt->n; j++) \
- if ( ptr[j]==vector_end ) break; \
+ if ( convert(&ptr[j * sizeof(type_t)])==vector_end ) break; \
if ( j==fmt->n ) \
{ \
/* all fields were used */ \
break; \
} \
if ( max_ploidy < j ) max_ploidy = j; \
- ptr += fmt->n; \
+ ptr += fmt->n * sizeof(type_t); \
} \
}
switch (fmt->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: error("Unexpected case: %d\n", fmt->type);
}
#undef BRANCH
int *map = ma->buf[i].rec[ma->buf[i].cur].map;
double *allele_prob = ma->tmpd;
int *idx = ma->tmpi;
- #define BRANCH(src_type_t, src_is_missing, src_is_vector_end, pl2prob_idx) { \
- src_type_t *src = (src_type_t*) fmt_ori->p; \
+ #define BRANCH(src_type_t, convert, src_is_missing, src_is_vector_end, pl2prob_idx) { \
+ uint8_t *src = fmt_ori->p; \
for (j=0; j<nsmpl; j++) \
{ \
for (k=0; k<line->n_allele; k++) allele_prob[k] = 0; \
for (k=0; k<line->n_allele; k++) \
for (l=0; l<=k; l++) \
{ \
- if ( src_is_missing || src_is_vector_end ) { src++; continue; } \
- double prob = ma->pl2prob[pl2prob_idx]; \
- allele_prob[k] += prob; \
- allele_prob[l] += prob; \
- src++; \
+ src_type_t val = convert(src); \
+ if ( !(src_is_missing) && !(src_is_vector_end) ) \
+ { \
+ double prob = ma->pl2prob[pl2prob_idx]; \
+ allele_prob[k] += prob; \
+ allele_prob[l] += prob; \
+ } \
+ src += sizeof(src_type_t); \
} \
/* insertion sort by allele probability, descending order, with the twist that REF (idx=0) always comes first */ \
allele_prob++; idx[0] = -1; idx++; /* keep REF first */ \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH( int8_t, *src==bcf_int8_missing, *src==bcf_int8_vector_end, *src); break;
- case BCF_BT_INT16: BRANCH(int16_t, *src==bcf_int16_missing, *src==bcf_int16_vector_end, *src>=0 && *src<PL2PROB_MAX ? *src : PL2PROB_MAX-1); break;
- case BCF_BT_INT32: BRANCH(int32_t, *src==bcf_int32_missing, *src==bcf_int32_vector_end, *src>=0 && *src<PL2PROB_MAX ? *src : PL2PROB_MAX-1); break;
+ case BCF_BT_INT8: BRANCH( int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, val); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, val>=0 && val<PL2PROB_MAX ? val : PL2PROB_MAX-1); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, val>=0 && val<PL2PROB_MAX ? val : PL2PROB_MAX-1); break;
default: error("Unexpected case: %d, PL\n", fmt_ori->type);
}
#undef BRANCH
continue;
}
- #define BRANCH(type_t, vector_end) { \
- type_t *p_ori = (type_t*) fmt_ori->p; \
+ #define BRANCH(type_t, convert, vector_end) { \
+ uint8_t *p_ori = fmt_ori->p; \
if ( !ma->buf[i].rec[irec].als_differ ) \
{ \
/* the allele numbering is unchanged */ \
{ \
for (k=0; k<fmt_ori->n; k++) \
{ \
- if ( p_ori[k]==vector_end ) break; /* smaller ploidy */ \
+ type_t val = convert(&p_ori[k * sizeof(type_t)]); \
+ if ( val==vector_end ) break; /* smaller ploidy */ \
ma->smpl_ploidy[ismpl+j]++; \
- if ( bcf_gt_is_missing(p_ori[k]) ) tmp[k] = 0; /* missing allele */ \
- else tmp[k] = p_ori[k]; \
+ if ( bcf_gt_is_missing(val) ) tmp[k] = 0; /* missing allele */ \
+ else tmp[k] = val; \
} \
for (; k<nsize; k++) tmp[k] = bcf_int32_vector_end; \
tmp += nsize; \
- p_ori += fmt_ori->n; \
+ p_ori += fmt_ori->n * sizeof(type_t); \
} \
ismpl += bcf_hdr_nsamples(hdr); \
continue; \
{ \
for (k=0; k<fmt_ori->n; k++) \
{ \
- if ( p_ori[k]==vector_end ) break; /* smaller ploidy */ \
+ type_t val = convert(&p_ori[k * sizeof(type_t)]); \
+ if ( val==vector_end ) break; /* smaller ploidy */ \
ma->smpl_ploidy[ismpl+j]++; \
- if ( bcf_gt_is_missing(p_ori[k]) ) tmp[k] = 0; /* missing allele */ \
+ if ( bcf_gt_is_missing(val) ) tmp[k] = 0; /* missing allele */ \
else \
{ \
- int al = (p_ori[k]>>1) - 1; \
+ int al = (val>>1) - 1; \
al = al<=0 ? al + 1 : ma->buf[i].rec[irec].map[al] + 1; \
- tmp[k] = (al << 1) | ((p_ori[k])&1); \
+ tmp[k] = (al << 1) | ((val)&1); \
} \
} \
for (; k<nsize; k++) tmp[k] = bcf_int32_vector_end; \
tmp += nsize; \
- p_ori += fmt_ori->n; \
+ p_ori += fmt_ori->n * sizeof(type_t); \
} \
ismpl += bcf_hdr_nsamples(hdr); \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: error("Unexpected case: %d\n", fmt_ori->type);
}
#undef BRANCH
if ( 2*fmt_ori->n!=line->n_allele*(line->n_allele+1) ) error("Todo: localization of missing or haploid Number=G tags\n");
// localize
- #define BRANCH(tgt_type_t, src_type_t, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
+ #define BRANCH(tgt_type_t, src_type_t, convert, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
for (j=0; j<nsmpl; j++) \
{ \
- src_type_t *src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
+ uint8_t *src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
tgt_type_t *tgt = (tgt_type_t *) ma->tmp_arr + ismpl*nsize; \
int *laa = ma->laa + (1+args->local_alleles)*ismpl; \
int ii,ij,tgt_idx = 0; \
for (ij=0; ij<=ii; ij++) \
{ \
int src_idx = bcf_alleles2gt(laa[ii],laa[ij]); \
+ src_type_t val = convert(&src[src_idx * sizeof(src_type_t)]); \
if ( src_is_missing ) tgt_set_missing; \
else if ( src_is_vector_end ) break; \
- else tgt[tgt_idx] = src[src_idx]; \
+ else tgt[tgt_idx] = val; \
tgt_idx++; \
} \
} \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH(int32_t, int8_t, src[src_idx]==bcf_int8_missing, src[src_idx]==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT16: BRANCH(int32_t, int16_t, src[src_idx]==bcf_int16_missing, src[src_idx]==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, int32_t, src[src_idx]==bcf_int32_missing, src[src_idx]==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_FLOAT: BRANCH(float, float, bcf_float_is_missing(src[src_idx]), bcf_float_is_vector_end(src[src_idx]), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
+ case BCF_BT_INT8: BRANCH(int32_t, int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int32_t, int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, int32_t, le_to_i16, val==bcf_int32_missing, val==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_FLOAT: BRANCH(float, float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
default: error("Unexpected case: %d, %s\n", fmt_ori->type, key);
}
#undef BRANCH
}
// localize
- #define BRANCH(tgt_type_t, src_type_t, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
+ #define BRANCH(tgt_type_t, src_type_t, convert, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
for (j=0; j<nsmpl; j++) \
{ \
- src_type_t *src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
+ uint8_t *src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
tgt_type_t *tgt = (tgt_type_t *) ma->tmp_arr + ismpl*nsize; \
int *laa = ma->laa + (1+args->local_alleles)*ismpl; \
int ii,tgt_idx = 0; \
{ \
if ( laa[ii]==bcf_int32_missing || laa[ii]==bcf_int32_vector_end ) break; \
int src_idx = laa[ii] - ibeg; \
+ src_type_t val = convert(&src[src_idx * sizeof(src_type_t)]); \
if ( src_is_missing ) tgt_set_missing; \
else if ( src_is_vector_end ) break; \
- else tgt[tgt_idx] = src[src_idx]; \
+ else tgt[tgt_idx] = val; \
tgt_idx++; \
} \
if ( !tgt_idx ) { tgt_set_missing; tgt_idx++; } \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH(int32_t, int8_t, src[src_idx]==bcf_int8_missing, src[src_idx]==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT16: BRANCH(int32_t, int16_t, src[src_idx]==bcf_int16_missing, src[src_idx]==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, int32_t, src[src_idx]==bcf_int32_missing, src[src_idx]==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_FLOAT: BRANCH(float, float, bcf_float_is_missing(src[src_idx]), bcf_float_is_vector_end(src[src_idx]), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
+ case BCF_BT_INT8: BRANCH(int32_t, int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int32_t, int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_FLOAT: BRANCH(float, float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
default: error("Unexpected case: %d, %s\n", fmt_ori->type, key);
}
#undef BRANCH
}
// set the values
- #define BRANCH(tgt_type_t, src_type_t, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
+ #define BRANCH(tgt_type_t, src_type_t, convert, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
int j, l, k; \
tgt_type_t *tgt = (tgt_type_t *) ma->tmp_arr + ismpl*nsize; \
if ( !fmt_ori ) \
ismpl += bcf_hdr_nsamples(hdr); \
continue; \
} \
- src_type_t *src = (src_type_t*) fmt_ori->p; \
+ uint8_t *src = fmt_ori->p; \
if ( (length!=BCF_VL_G && length!=BCF_VL_A && length!=BCF_VL_R) || (line->n_allele==out->n_allele && !ma->buf[i].rec[irec].als_differ) ) \
{ \
/* alleles unchanged, copy over */ \
{ \
if ( src_is_vector_end ) break; \
else if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
- tgt++; src++; \
+ else *tgt = convert(src); \
+ tgt++; src += sizeof(src_type_t); \
} \
for (k=l; k<nsize; k++) { tgt_set_vector_end; tgt++; } \
- src += fmt_ori->n - l; \
+ src += sizeof(src_type_t) * (fmt_ori->n - l); \
} \
ismpl += bcf_hdr_nsamples(hdr); \
continue; \
for (j=0; j<bcf_hdr_nsamples(hdr); j++) \
{ \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize; \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
- if ( (src_is_missing && fmt_ori->n==1) || (++src && src_is_vector_end) ) \
+ src = fmt_ori->p + sizeof(src_type_t) * j * fmt_ori->n; \
+ int tag_missing = src_is_missing && fmt_ori->n==1; \
+ if ( src_is_missing && fmt_ori->n>1 ) { \
+ src += sizeof(src_type_t); \
+ tag_missing = src_is_vector_end ; \
+ } \
+ if ( tag_missing ) \
{ \
/* tag with missing value "." */ \
tgt_set_missing; \
int ngsize = haploid ? out->n_allele : out->n_allele*(out->n_allele + 1)/2; \
if ( ma->buf[i].unkn_allele ) /* Use value from the unknown allele when available */ \
{ \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
+ src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
int iunkn = haploid ? ma->buf[i].unkn_allele : (ma->buf[i].unkn_allele+1)*(ma->buf[i].unkn_allele + 2)/2 - 1; \
- for (l=0; l<ngsize; l++) { *tgt = src[iunkn]; tgt++; } \
+ src_type_t val = convert(&src[iunkn * sizeof(src_type_t)]); \
+ for (l=0; l<ngsize; l++) { *tgt = val; tgt++; } \
} \
else if ( mrule && mrule->type==MERGE_MISSING_CONST ) \
{ \
} \
else if ( mrule && mrule->type==MERGE_MISSING_MAX ) \
{ \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
- src_type_t max = src[0]; \
- for (l=1; l<fmt_ori->n; l++) if ( max < src[l] ) max = src[l]; \
+ src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
+ src_type_t max = convert(src); \
+ for (l=1; l<fmt_ori->n; l++) \
+ { \
+ src_type_t val = convert(&src[l * sizeof(src_type_t)]); \
+ if ( max < val ) max = val; \
+ } \
for (l=0; l<ngsize; l++) { *tgt = max; tgt++; } \
} \
else \
for (iori=0; iori<line->n_allele; iori++) \
{ \
inew = ma->buf[i].rec[irec].map[iori]; \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n + iori; \
+ src = fmt_ori->p + (j*fmt_ori->n + iori) * sizeof(src_type_t); \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize + inew; \
if ( src_is_vector_end ) break; \
if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
+ else *tgt = convert(src); \
} \
} \
else \
jnew = ma->buf[i].rec[irec].map[jori]; \
int kori = iori*(iori+1)/2 + jori; \
int knew = inew>jnew ? inew*(inew+1)/2 + jnew : jnew*(jnew+1)/2 + inew; \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n + kori; \
+ src = fmt_ori->p + (j*fmt_ori->n + kori) * sizeof(src_type_t); \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize + knew; \
if ( src_is_vector_end ) \
{ \
break; \
} \
if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
+ else *tgt = convert(src); \
} \
} \
} \
for (j=0; j<bcf_hdr_nsamples(hdr); j++) \
{ \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize; \
- src = (src_type_t*) (fmt_ori->p + j*fmt_ori->size); \
- if ( (src_is_missing && fmt_ori->n==1) || (++src && src_is_vector_end) ) \
+ src = fmt_ori->p + j*fmt_ori->size; \
+ int tag_missing = src_is_missing && fmt_ori->n==1; \
+ if ( src_is_missing && fmt_ori->n>1 ) { \
+ src += sizeof(src_type_t); \
+ tag_missing = src_is_vector_end ; \
+ } \
+ if ( tag_missing ) \
{ \
/* tag with missing value "." */ \
tgt_set_missing; \
for (l=1; l<nsize; l++) { tgt++; tgt_set_vector_end; } \
continue; \
} \
- src = (src_type_t*) (fmt_ori->p + j*fmt_ori->size); \
+ src = fmt_ori->p + j*fmt_ori->size; \
if ( ma->buf[i].unkn_allele ) /* Use value from the unknown allele when available */ \
{ \
int iunkn = ma->buf[i].unkn_allele; \
- for (l=0; l<nsize; l++) { *tgt = src[iunkn]; tgt++; } \
+ src_type_t val = convert(&src[iunkn * sizeof(src_type_t)]); \
+ for (l=0; l<nsize; l++) { *tgt = val; tgt++; } \
} \
else if ( mrule && mrule->type==MERGE_MISSING_CONST ) \
{ \
} \
else if ( mrule && mrule->type==MERGE_MISSING_MAX ) \
{ \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
- src_type_t max = src[0]; \
- for (l=1; l<fmt_ori->n; l++) if ( max < src[l] ) max = src[l]; \
+ src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
+ src_type_t max = convert(src); \
+ for (l=1; l<fmt_ori->n; l++) \
+ { \
+ src_type_t val = convert(&src[l * sizeof(src_type_t)]); \
+ if ( max < val ) max = val; \
+ } \
for (l=0; l<nsize; l++) { *tgt = max; tgt++; } \
} \
else \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize + inew; \
if ( src_is_vector_end ) break; \
if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
- src++; \
+ else *tgt = convert(src); \
+ src += sizeof(src_type_t); \
} \
} \
} \
}
switch (type)
{
- case BCF_BT_INT8: BRANCH(int32_t, int8_t, *src==bcf_int8_missing, *src==bcf_int8_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
- case BCF_BT_INT16: BRANCH(int32_t, int16_t, *src==bcf_int16_missing, *src==bcf_int16_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, int32_t, *src==bcf_int32_missing, *src==bcf_int32_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
- case BCF_BT_FLOAT: BRANCH(float, float, bcf_float_is_missing(*src), bcf_float_is_vector_end(*src), bcf_float_set_missing(*tgt), bcf_float_set_vector_end(*tgt)); break;
+ case BCF_BT_INT8: BRANCH(int32_t, int8_t, le_to_i8, le_to_i8(src)==bcf_int8_missing, le_to_i8(src)==bcf_int8_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int32_t, int16_t, le_to_i16, le_to_i16(src)==bcf_int16_missing, le_to_i16(src)==bcf_int16_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, int32_t, le_to_i32, le_to_i32(src)==bcf_int32_missing, le_to_i32(src)==bcf_int32_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
+ case BCF_BT_FLOAT: BRANCH(float, float, le_to_float, bcf_float_is_missing(le_to_float(src)), bcf_float_is_vector_end(le_to_float(src)), bcf_float_set_missing(*tgt), bcf_float_set_vector_end(*tgt)); break;
default: error("Unexpected case: %d, %s\n", type, key);
}
#undef BRANCH
}
else
bcf_update_info_int32(args->out_hdr, out, "END", NULL, 0);
+
+ int iunseen;
+ if ( args->trim_star_allele && (out->n_allele > 2 || args->trim_star_allele > 1) && (iunseen=get_unseen_allele(out)) && iunseen>0 )
+ {
+ // the unobserved star allele should be trimmed, either it is variant site or trimming of all sites was requested
+ kbitset_t *rm_set = kbs_init(out->n_allele);
+ kbs_insert(rm_set, iunseen);
+ if ( bcf_remove_allele_set(args->out_hdr,out,rm_set) )
+ error("[%s] Error: failed to trim the unobserved allele at %s:%"PRIhts_pos"\n",__func__,bcf_seqname(args->out_hdr,out),out->pos+1);
+ kbs_destroy(rm_set);
+ }
if ( bcf_write1(args->out_fh, args->out_hdr, out)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
bcf_clear1(out);
-
// Inactivate blocks which do not extend beyond END and find new gvcf_min
min = INT_MAX;
for (i=0; i<args->files->nreaders; i++)
if ( args->do_gvcf )
bcf_update_info_int32(args->out_hdr, out, "END", NULL, 0);
merge_format(args, out);
+ int iunseen;
+ if ( args->trim_star_allele && (out->n_allele > 2 || args->trim_star_allele > 1) && (iunseen=get_unseen_allele(out)) && iunseen>0 )
+ {
+ // the unobserved star allele should be trimmed, either it is variant site or trimming of all sites was requested
+ kbitset_t *rm_set = kbs_init(out->n_allele);
+ kbs_insert(rm_set, iunseen);
+ if ( bcf_remove_allele_set(args->out_hdr,out,rm_set) )
+ error("[%s] Error: failed to trim the unobserved allele at %s:%"PRIhts_pos"\n",__func__,bcf_seqname(args->out_hdr,out),out->pos+1);
+ kbs_destroy(rm_set);
+ }
if ( bcf_write1(args->out_fh, args->out_hdr, out)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
bcf_clear1(out);
}
if ( hts_close(args->out_fh)!=0 ) error("[%s] Error: close failed .. %s\n", __func__,args->output_fname);
return;
}
- else if ( args->write_index && init_index(args->out_fh,args->out_hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ else if ( init_index2(args->out_fh,args->out_hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
- if ( args->collapse==COLLAPSE_NONE ) args->vcmp = vcmp_init();
+ args->vcmp = vcmp_init();
args->maux = maux_init(args);
args->out_line = bcf_init1();
args->tmph = kh_init(strdict);
fprintf(stderr, "Usage: bcftools merge [options] <A.vcf.gz> <B.vcf.gz> [...]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Options:\n");
+ fprintf(stderr, " --force-no-index Merge unindexed files, synonymous to --no-index\n");
fprintf(stderr, " --force-samples Resolve duplicate sample names\n");
+ fprintf(stderr, " --force-single Run even if there is only one file on input\n");
fprintf(stderr, " --print-header Print only the merged header and exit\n");
fprintf(stderr, " --use-header FILE Use the provided header\n");
fprintf(stderr, " -0 --missing-to-ref Assume genotypes at missing sites are 0/0\n");
fprintf(stderr, " -f, --apply-filters LIST Require at least one of the listed FILTER strings (e.g. \"PASS,.\")\n");
fprintf(stderr, " -F, --filter-logic x|+ Remove filters if some input is PASS (\"x\"), or apply all filters (\"+\") [+]\n");
- fprintf(stderr, " -g, --gvcf -|REF.FA Merge gVCF blocks, INFO/END tag is expected. Implies -i QS:sum,MinDP:min,I16:sum,IDV:max,IMF:max -M PL:max,AD:0\n");
+ fprintf(stderr, " -g, --gvcf -|REF.FA Merge gVCF blocks, INFO/END tag is expected. Implies -i QS:sum,MinDP:min,MIN_DP:min,I16:sum,IDV:max,IMF:max -M PL:max,AD:0\n");
fprintf(stderr, " -i, --info-rules TAG:METHOD,.. Rules for merging INFO fields (method is one of sum,avg,min,max,join) or \"-\" to turn off the default [DP:sum,DP4:sum]\n");
fprintf(stderr, " -l, --file-list FILE Read file names from the file\n");
- fprintf(stderr, " -L, --local-alleles INT EXPERIMENTAL: if more than <int> ALT alleles are encountered, drop FMT/PL and output LAA+LPL instead; 0=unlimited [0]\n");
- fprintf(stderr, " -m, --merge STRING Allow multiallelic records for <snps|indels|both|snp-ins-del|all|none|id>, see man page for details [both]\n");
+ fprintf(stderr, " -L, --local-alleles INT If more than INT alt alleles are encountered, drop FMT/PL and output LAA+LPL instead; 0=unlimited [0]\n");
+ fprintf(stderr, " -m, --merge STRING[*|**] Allow multiallelic records for snps,indels,both,snp-ins-del,all,none,id,*,**; see man page for details [both]\n");
fprintf(stderr, " -M, --missing-rules TAG:METHOD Rules for replacing missing values in numeric vectors (.,0,max) when unknown allele <*> is not present [.]\n");
fprintf(stderr, " --no-index Merge unindexed files, the same chromosomal order is required and -r/-R are not allowed\n");
fprintf(stderr, " --no-version Do not append version and command line to the header\n");
fprintf(stderr, " -r, --regions REGION Restrict to comma-separated list of regions\n");
fprintf(stderr, " -R, --regions-file FILE Restrict to regions listed in a file\n");
fprintf(stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
- fprintf(stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
exit(1);
}
{"missing-rules",required_argument,NULL,'M'},
{"no-version",no_argument,NULL,8},
{"no-index",no_argument,NULL,10},
+ {"force-no-index",no_argument,NULL,10},
+ {"force-single",no_argument,NULL,12},
{"filter-logic",required_argument,NULL,'F'},
- {"write-index",no_argument,NULL,11},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hm:f:r:R:o:O:i:M:l:g:F:0L:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hm:f:r:R:o:O:i:M:l:g:F:0L:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'L':
args->local_alleles = strtol(optarg,&tmp,10);
}
break;
case 'm':
+ {
+ int len = strlen(optarg);
+ if ( optarg[len-1]=='*' ) { args->trim_star_allele++; len--; }
+ if ( optarg[len-1]=='*' ) { args->trim_star_allele++; len--; }
+ if ( optarg[len-1]==',' ) len--;
args->collapse = COLLAPSE_NONE;
- if ( !strcmp(optarg,"snps") ) args->collapse |= COLLAPSE_SNPS;
- else if ( !strcmp(optarg,"indels") ) args->collapse |= COLLAPSE_INDELS;
- else if ( !strcmp(optarg,"both") ) args->collapse |= COLLAPSE_BOTH;
- else if ( !strcmp(optarg,"any") ) args->collapse |= COLLAPSE_ANY;
- else if ( !strcmp(optarg,"all") ) args->collapse |= COLLAPSE_ANY;
- else if ( !strcmp(optarg,"none") ) args->collapse = COLLAPSE_NONE;
- else if ( !strcmp(optarg,"snp-ins-del") ) args->collapse = COLLAPSE_SNP_INS_DEL|COLLAPSE_SNPS;
- else if ( !strcmp(optarg,"id") ) { args->collapse = COLLAPSE_NONE; args->merge_by_id = 1; }
+ if ( !strncmp(optarg,"snp-ins-del",len) ) args->collapse = COLLAPSE_SNP_INS_DEL|COLLAPSE_SNPS;
+ else if ( !strncmp(optarg,"snps",len) ) args->collapse |= COLLAPSE_SNPS;
+ else if ( !strncmp(optarg,"indels",len) ) args->collapse |= COLLAPSE_INDELS;
+ else if ( !strncmp(optarg,"id",len) ) { args->collapse = COLLAPSE_NONE; args->merge_by_id = 1; }
+ else if ( !strncmp(optarg,"any",len) ) args->collapse |= COLLAPSE_ANY;
+ else if ( !strncmp(optarg,"all",len) ) args->collapse |= COLLAPSE_ANY;
+ else if ( !strncmp(optarg,"both",len) ) args->collapse |= COLLAPSE_BOTH;
+ else if ( !strncmp(optarg,"none",len) ) args->collapse = COLLAPSE_NONE;
else error("The -m type \"%s\" is not recognised.\n", optarg);
break;
+ }
case 'f': args->files->apply_filters = optarg; break;
case 'r': args->regions_list = optarg; break;
case 'R': args->regions_list = optarg; regions_is_file = 1; break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
case 10 : args->no_index = 1; break;
- case 11 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
+ case 12 : args->force_single = 1; break;
case 'h':
case '?': usage(); break;
default: error("Unknown argument: %s\n", optarg);
}
}
if ( argc==optind && !args->file_list ) usage();
- if ( argc-optind<2 && !args->file_list ) usage();
-
if ( args->no_index )
{
if ( args->regions_list ) error("Error: cannot combine --no-index with -r/-R\n");
for (i=0; i<nfiles; i++) free(files[i]);
free(files);
}
+ if ( !args->files->nreaders ) usage();
+ if ( args->files->nreaders==1 && !args->force_single ) error("Expected two or more files to merge, got only one. Use --force-single to proceed anyway\n");
+
merge_vcf(args);
bcf_sr_destroy(args->files);
if ( args->regs ) regidx_destroy(args->regs);
/* vcfmerge.c -- Merge multiple VCF/BCF files to create one multi-sample file.
- Copyright (C) 2012-2023 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/faidx.h>
+#include <htslib/kbitset.h>
+#include <htslib/hts_endian.h>
#include <math.h>
#include <ctype.h>
#include <time.h>
maux_t *maux;
regidx_t *regs; // apply regions only after the blocks are expanded
regitr_t *regs_itr;
- int header_only, collapse, output_type, force_samples, merge_by_id, do_gvcf, filter_logic, missing_to_ref, no_index;
+ int header_only, collapse, output_type, force_samples, force_single, merge_by_id, do_gvcf, filter_logic, missing_to_ref, no_index;
char *header_fname, *output_fname, *regions_list, *info_rules, *file_list;
faidx_t *gvcf_fai;
info_rule_t *rules;
int keep_AC_AN;
char *index_fn;
int write_index;
+ int trim_star_allele; // 0=don't trim; 1=trim at variant sites; 2=trim at all sites
}
args_t;
if ( str.l ) kputc(',',&str);
kputs("QS:sum",&str);
}
+ if ( args->do_gvcf && bcf_hdr_idinfo_exists(args->out_hdr,BCF_HL_INFO,bcf_hdr_id2int(args->out_hdr, BCF_DT_ID, "MIN_DP")) )
+ {
+ if ( str.l ) kputc(',',&str);
+ kputs("MIN_DP:min",&str);
+ }
if ( args->do_gvcf && bcf_hdr_idinfo_exists(args->out_hdr,BCF_HL_INFO,bcf_hdr_id2int(args->out_hdr, BCF_DT_ID, "MinDP")) )
{
if ( str.l ) kputc(',',&str);
if ( len==BCF_VL_A || len==BCF_VL_R )
{
int ifrom = len==BCF_VL_A ? 1 : 0;
- #define BRANCH(type_t, is_missing, is_vector_end, out_type_t) { \
- type_t *src = (type_t *) info->vptr; \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, out_type_t) { \
+ uint8_t *src = info->vptr; \
out_type_t *tgt = (out_type_t *) agr->buf; \
int iori, inew; \
- for (iori=ifrom; iori<line->n_allele; iori++) \
+ for (iori=ifrom; iori<line->n_allele; iori++, src += sizeof(type_t)) \
{ \
+ type_t val = convert(src); \
if ( is_vector_end ) break; \
if ( is_missing ) continue; \
inew = als->map[iori] - ifrom; \
- tgt[inew] = *src; \
- src++; \
+ tgt[inew] = val; \
} \
}
switch (info->type) {
- case BCF_BT_INT8: BRANCH(int8_t, *src==bcf_int8_missing, *src==bcf_int8_vector_end, int); break;
- case BCF_BT_INT16: BRANCH(int16_t, *src==bcf_int16_missing, *src==bcf_int16_vector_end, int); break;
- case BCF_BT_INT32: BRANCH(int32_t, *src==bcf_int32_missing, *src==bcf_int32_vector_end, int); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(*src), bcf_float_is_vector_end(*src), float); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, int); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, int); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, int); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), float); break;
default: fprintf(bcftools_stderr,"TODO: %s:%d .. info->type=%d\n", __FILE__,__LINE__, info->type); bcftools_exit(1);
}
#undef BRANCH
}
else
{
- #define BRANCH(type_t, is_missing, is_vector_end, out_type_t) { \
- type_t *src = (type_t *) info->vptr; \
+ #define BRANCH(type_t, convert, is_missing, is_vector_end, out_type_t) { \
+ uint8_t *src = info->vptr; \
out_type_t *tgt = (out_type_t *) agr->buf; \
int iori,jori, inew,jnew; \
for (iori=0; iori<line->n_allele; iori++) \
{ \
jnew = als->map[jori]; \
int kori = iori*(iori+1)/2 + jori; \
+ type_t val = convert(&src[kori * sizeof(type_t)]); \
if ( is_vector_end ) break; \
if ( is_missing ) continue; \
int knew = inew>jnew ? inew*(inew+1)/2 + jnew : jnew*(jnew+1)/2 + inew; \
- tgt[knew] = src[kori]; \
+ tgt[knew] = val; \
} \
if ( jori<=iori ) break; \
} \
}
switch (info->type) {
- case BCF_BT_INT8: BRANCH(int8_t, src[kori]==bcf_int8_missing, src[kori]==bcf_int8_vector_end, int); break;
- case BCF_BT_INT16: BRANCH(int16_t, src[kori]==bcf_int16_missing, src[kori]==bcf_int16_vector_end, int); break;
- case BCF_BT_INT32: BRANCH(int32_t, src[kori]==bcf_int32_missing, src[kori]==bcf_int32_vector_end, int); break;
- case BCF_BT_FLOAT: BRANCH(float, bcf_float_is_missing(src[kori]), bcf_float_is_vector_end(src[kori]), float); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, int); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, int); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, int); break;
+ case BCF_BT_FLOAT: BRANCH(float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), float); break;
default: fprintf(bcftools_stderr,"TODO: %s:%d .. info->type=%d\n", __FILE__,__LINE__, info->type); bcftools_exit(1);
}
#undef BRANCH
{
int i,j, max_ploidy = 0;
- #define BRANCH(type_t, vector_end) { \
- type_t *ptr = (type_t*) fmt->p; \
+ #define BRANCH(type_t, convert, vector_end) { \
+ uint8_t *ptr = fmt->p; \
for (i=0; i<nsmpl; i++) \
{ \
for (j=0; j<fmt->n; j++) \
- if ( ptr[j]==vector_end ) break; \
+ if ( convert(&ptr[j * sizeof(type_t)])==vector_end ) break; \
if ( j==fmt->n ) \
{ \
/* all fields were used */ \
break; \
} \
if ( max_ploidy < j ) max_ploidy = j; \
- ptr += fmt->n; \
+ ptr += fmt->n * sizeof(type_t); \
} \
}
switch (fmt->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: error("Unexpected case: %d\n", fmt->type);
}
#undef BRANCH
int *map = ma->buf[i].rec[ma->buf[i].cur].map;
double *allele_prob = ma->tmpd;
int *idx = ma->tmpi;
- #define BRANCH(src_type_t, src_is_missing, src_is_vector_end, pl2prob_idx) { \
- src_type_t *src = (src_type_t*) fmt_ori->p; \
+ #define BRANCH(src_type_t, convert, src_is_missing, src_is_vector_end, pl2prob_idx) { \
+ uint8_t *src = fmt_ori->p; \
for (j=0; j<nsmpl; j++) \
{ \
for (k=0; k<line->n_allele; k++) allele_prob[k] = 0; \
for (k=0; k<line->n_allele; k++) \
for (l=0; l<=k; l++) \
{ \
- if ( src_is_missing || src_is_vector_end ) { src++; continue; } \
- double prob = ma->pl2prob[pl2prob_idx]; \
- allele_prob[k] += prob; \
- allele_prob[l] += prob; \
- src++; \
+ src_type_t val = convert(src); \
+ if ( !(src_is_missing) && !(src_is_vector_end) ) \
+ { \
+ double prob = ma->pl2prob[pl2prob_idx]; \
+ allele_prob[k] += prob; \
+ allele_prob[l] += prob; \
+ } \
+ src += sizeof(src_type_t); \
} \
/* insertion sort by allele probability, descending order, with the twist that REF (idx=0) always comes first */ \
allele_prob++; idx[0] = -1; idx++; /* keep REF first */ \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH( int8_t, *src==bcf_int8_missing, *src==bcf_int8_vector_end, *src); break;
- case BCF_BT_INT16: BRANCH(int16_t, *src==bcf_int16_missing, *src==bcf_int16_vector_end, *src>=0 && *src<PL2PROB_MAX ? *src : PL2PROB_MAX-1); break;
- case BCF_BT_INT32: BRANCH(int32_t, *src==bcf_int32_missing, *src==bcf_int32_vector_end, *src>=0 && *src<PL2PROB_MAX ? *src : PL2PROB_MAX-1); break;
+ case BCF_BT_INT8: BRANCH( int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, val); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, val>=0 && val<PL2PROB_MAX ? val : PL2PROB_MAX-1); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, val>=0 && val<PL2PROB_MAX ? val : PL2PROB_MAX-1); break;
default: error("Unexpected case: %d, PL\n", fmt_ori->type);
}
#undef BRANCH
continue;
}
- #define BRANCH(type_t, vector_end) { \
- type_t *p_ori = (type_t*) fmt_ori->p; \
+ #define BRANCH(type_t, convert, vector_end) { \
+ uint8_t *p_ori = fmt_ori->p; \
if ( !ma->buf[i].rec[irec].als_differ ) \
{ \
/* the allele numbering is unchanged */ \
{ \
for (k=0; k<fmt_ori->n; k++) \
{ \
- if ( p_ori[k]==vector_end ) break; /* smaller ploidy */ \
+ type_t val = convert(&p_ori[k * sizeof(type_t)]); \
+ if ( val==vector_end ) break; /* smaller ploidy */ \
ma->smpl_ploidy[ismpl+j]++; \
- if ( bcf_gt_is_missing(p_ori[k]) ) tmp[k] = 0; /* missing allele */ \
- else tmp[k] = p_ori[k]; \
+ if ( bcf_gt_is_missing(val) ) tmp[k] = 0; /* missing allele */ \
+ else tmp[k] = val; \
} \
for (; k<nsize; k++) tmp[k] = bcf_int32_vector_end; \
tmp += nsize; \
- p_ori += fmt_ori->n; \
+ p_ori += fmt_ori->n * sizeof(type_t); \
} \
ismpl += bcf_hdr_nsamples(hdr); \
continue; \
{ \
for (k=0; k<fmt_ori->n; k++) \
{ \
- if ( p_ori[k]==vector_end ) break; /* smaller ploidy */ \
+ type_t val = convert(&p_ori[k * sizeof(type_t)]); \
+ if ( val==vector_end ) break; /* smaller ploidy */ \
ma->smpl_ploidy[ismpl+j]++; \
- if ( bcf_gt_is_missing(p_ori[k]) ) tmp[k] = 0; /* missing allele */ \
+ if ( bcf_gt_is_missing(val) ) tmp[k] = 0; /* missing allele */ \
else \
{ \
- int al = (p_ori[k]>>1) - 1; \
+ int al = (val>>1) - 1; \
al = al<=0 ? al + 1 : ma->buf[i].rec[irec].map[al] + 1; \
- tmp[k] = (al << 1) | ((p_ori[k])&1); \
+ tmp[k] = (al << 1) | ((val)&1); \
} \
} \
for (; k<nsize; k++) tmp[k] = bcf_int32_vector_end; \
tmp += nsize; \
- p_ori += fmt_ori->n; \
+ p_ori += fmt_ori->n * sizeof(type_t); \
} \
ismpl += bcf_hdr_nsamples(hdr); \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH(int8_t, bcf_int8_vector_end); break;
- case BCF_BT_INT16: BRANCH(int16_t, bcf_int16_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, bcf_int32_vector_end); break;
+ case BCF_BT_INT8: BRANCH(int8_t, le_to_i8, bcf_int8_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int16_t, le_to_i16, bcf_int16_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, le_to_i32, bcf_int32_vector_end); break;
default: error("Unexpected case: %d\n", fmt_ori->type);
}
#undef BRANCH
if ( 2*fmt_ori->n!=line->n_allele*(line->n_allele+1) ) error("Todo: localization of missing or haploid Number=G tags\n");
// localize
- #define BRANCH(tgt_type_t, src_type_t, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
+ #define BRANCH(tgt_type_t, src_type_t, convert, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
for (j=0; j<nsmpl; j++) \
{ \
- src_type_t *src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
+ uint8_t *src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
tgt_type_t *tgt = (tgt_type_t *) ma->tmp_arr + ismpl*nsize; \
int *laa = ma->laa + (1+args->local_alleles)*ismpl; \
int ii,ij,tgt_idx = 0; \
for (ij=0; ij<=ii; ij++) \
{ \
int src_idx = bcf_alleles2gt(laa[ii],laa[ij]); \
+ src_type_t val = convert(&src[src_idx * sizeof(src_type_t)]); \
if ( src_is_missing ) tgt_set_missing; \
else if ( src_is_vector_end ) break; \
- else tgt[tgt_idx] = src[src_idx]; \
+ else tgt[tgt_idx] = val; \
tgt_idx++; \
} \
} \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH(int32_t, int8_t, src[src_idx]==bcf_int8_missing, src[src_idx]==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT16: BRANCH(int32_t, int16_t, src[src_idx]==bcf_int16_missing, src[src_idx]==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, int32_t, src[src_idx]==bcf_int32_missing, src[src_idx]==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_FLOAT: BRANCH(float, float, bcf_float_is_missing(src[src_idx]), bcf_float_is_vector_end(src[src_idx]), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
+ case BCF_BT_INT8: BRANCH(int32_t, int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int32_t, int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, int32_t, le_to_i16, val==bcf_int32_missing, val==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_FLOAT: BRANCH(float, float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
default: error("Unexpected case: %d, %s\n", fmt_ori->type, key);
}
#undef BRANCH
}
// localize
- #define BRANCH(tgt_type_t, src_type_t, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
+ #define BRANCH(tgt_type_t, src_type_t, convert, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
for (j=0; j<nsmpl; j++) \
{ \
- src_type_t *src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
+ uint8_t *src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
tgt_type_t *tgt = (tgt_type_t *) ma->tmp_arr + ismpl*nsize; \
int *laa = ma->laa + (1+args->local_alleles)*ismpl; \
int ii,tgt_idx = 0; \
{ \
if ( laa[ii]==bcf_int32_missing || laa[ii]==bcf_int32_vector_end ) break; \
int src_idx = laa[ii] - ibeg; \
+ src_type_t val = convert(&src[src_idx * sizeof(src_type_t)]); \
if ( src_is_missing ) tgt_set_missing; \
else if ( src_is_vector_end ) break; \
- else tgt[tgt_idx] = src[src_idx]; \
+ else tgt[tgt_idx] = val; \
tgt_idx++; \
} \
if ( !tgt_idx ) { tgt_set_missing; tgt_idx++; } \
}
switch (fmt_ori->type)
{
- case BCF_BT_INT8: BRANCH(int32_t, int8_t, src[src_idx]==bcf_int8_missing, src[src_idx]==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT16: BRANCH(int32_t, int16_t, src[src_idx]==bcf_int16_missing, src[src_idx]==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, int32_t, src[src_idx]==bcf_int32_missing, src[src_idx]==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
- case BCF_BT_FLOAT: BRANCH(float, float, bcf_float_is_missing(src[src_idx]), bcf_float_is_vector_end(src[src_idx]), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
+ case BCF_BT_INT8: BRANCH(int32_t, int8_t, le_to_i8, val==bcf_int8_missing, val==bcf_int8_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int32_t, int16_t, le_to_i16, val==bcf_int16_missing, val==bcf_int16_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, int32_t, le_to_i32, val==bcf_int32_missing, val==bcf_int32_vector_end, tgt[tgt_idx]=bcf_int32_missing, tgt[tgt_idx]=bcf_int32_vector_end); break;
+ case BCF_BT_FLOAT: BRANCH(float, float, le_to_float, bcf_float_is_missing(val), bcf_float_is_vector_end(val), bcf_float_set_missing(tgt[tgt_idx]), bcf_float_set_vector_end(tgt[tgt_idx])); break;
default: error("Unexpected case: %d, %s\n", fmt_ori->type, key);
}
#undef BRANCH
}
// set the values
- #define BRANCH(tgt_type_t, src_type_t, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
+ #define BRANCH(tgt_type_t, src_type_t, convert, src_is_missing, src_is_vector_end, tgt_set_missing, tgt_set_vector_end) { \
int j, l, k; \
tgt_type_t *tgt = (tgt_type_t *) ma->tmp_arr + ismpl*nsize; \
if ( !fmt_ori ) \
ismpl += bcf_hdr_nsamples(hdr); \
continue; \
} \
- src_type_t *src = (src_type_t*) fmt_ori->p; \
+ uint8_t *src = fmt_ori->p; \
if ( (length!=BCF_VL_G && length!=BCF_VL_A && length!=BCF_VL_R) || (line->n_allele==out->n_allele && !ma->buf[i].rec[irec].als_differ) ) \
{ \
/* alleles unchanged, copy over */ \
{ \
if ( src_is_vector_end ) break; \
else if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
- tgt++; src++; \
+ else *tgt = convert(src); \
+ tgt++; src += sizeof(src_type_t); \
} \
for (k=l; k<nsize; k++) { tgt_set_vector_end; tgt++; } \
- src += fmt_ori->n - l; \
+ src += sizeof(src_type_t) * (fmt_ori->n - l); \
} \
ismpl += bcf_hdr_nsamples(hdr); \
continue; \
for (j=0; j<bcf_hdr_nsamples(hdr); j++) \
{ \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize; \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
- if ( (src_is_missing && fmt_ori->n==1) || (++src && src_is_vector_end) ) \
+ src = fmt_ori->p + sizeof(src_type_t) * j * fmt_ori->n; \
+ int tag_missing = src_is_missing && fmt_ori->n==1; \
+ if ( src_is_missing && fmt_ori->n>1 ) { \
+ src += sizeof(src_type_t); \
+ tag_missing = src_is_vector_end ; \
+ } \
+ if ( tag_missing ) \
{ \
/* tag with missing value "." */ \
tgt_set_missing; \
int ngsize = haploid ? out->n_allele : out->n_allele*(out->n_allele + 1)/2; \
if ( ma->buf[i].unkn_allele ) /* Use value from the unknown allele when available */ \
{ \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
+ src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
int iunkn = haploid ? ma->buf[i].unkn_allele : (ma->buf[i].unkn_allele+1)*(ma->buf[i].unkn_allele + 2)/2 - 1; \
- for (l=0; l<ngsize; l++) { *tgt = src[iunkn]; tgt++; } \
+ src_type_t val = convert(&src[iunkn * sizeof(src_type_t)]); \
+ for (l=0; l<ngsize; l++) { *tgt = val; tgt++; } \
} \
else if ( mrule && mrule->type==MERGE_MISSING_CONST ) \
{ \
} \
else if ( mrule && mrule->type==MERGE_MISSING_MAX ) \
{ \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
- src_type_t max = src[0]; \
- for (l=1; l<fmt_ori->n; l++) if ( max < src[l] ) max = src[l]; \
+ src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
+ src_type_t max = convert(src); \
+ for (l=1; l<fmt_ori->n; l++) \
+ { \
+ src_type_t val = convert(&src[l * sizeof(src_type_t)]); \
+ if ( max < val ) max = val; \
+ } \
for (l=0; l<ngsize; l++) { *tgt = max; tgt++; } \
} \
else \
for (iori=0; iori<line->n_allele; iori++) \
{ \
inew = ma->buf[i].rec[irec].map[iori]; \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n + iori; \
+ src = fmt_ori->p + (j*fmt_ori->n + iori) * sizeof(src_type_t); \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize + inew; \
if ( src_is_vector_end ) break; \
if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
+ else *tgt = convert(src); \
} \
} \
else \
jnew = ma->buf[i].rec[irec].map[jori]; \
int kori = iori*(iori+1)/2 + jori; \
int knew = inew>jnew ? inew*(inew+1)/2 + jnew : jnew*(jnew+1)/2 + inew; \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n + kori; \
+ src = fmt_ori->p + (j*fmt_ori->n + kori) * sizeof(src_type_t); \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize + knew; \
if ( src_is_vector_end ) \
{ \
break; \
} \
if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
+ else *tgt = convert(src); \
} \
} \
} \
for (j=0; j<bcf_hdr_nsamples(hdr); j++) \
{ \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize; \
- src = (src_type_t*) (fmt_ori->p + j*fmt_ori->size); \
- if ( (src_is_missing && fmt_ori->n==1) || (++src && src_is_vector_end) ) \
+ src = fmt_ori->p + j*fmt_ori->size; \
+ int tag_missing = src_is_missing && fmt_ori->n==1; \
+ if ( src_is_missing && fmt_ori->n>1 ) { \
+ src += sizeof(src_type_t); \
+ tag_missing = src_is_vector_end ; \
+ } \
+ if ( tag_missing ) \
{ \
/* tag with missing value "." */ \
tgt_set_missing; \
for (l=1; l<nsize; l++) { tgt++; tgt_set_vector_end; } \
continue; \
} \
- src = (src_type_t*) (fmt_ori->p + j*fmt_ori->size); \
+ src = fmt_ori->p + j*fmt_ori->size; \
if ( ma->buf[i].unkn_allele ) /* Use value from the unknown allele when available */ \
{ \
int iunkn = ma->buf[i].unkn_allele; \
- for (l=0; l<nsize; l++) { *tgt = src[iunkn]; tgt++; } \
+ src_type_t val = convert(&src[iunkn * sizeof(src_type_t)]); \
+ for (l=0; l<nsize; l++) { *tgt = val; tgt++; } \
} \
else if ( mrule && mrule->type==MERGE_MISSING_CONST ) \
{ \
} \
else if ( mrule && mrule->type==MERGE_MISSING_MAX ) \
{ \
- src = (src_type_t*) fmt_ori->p + j*fmt_ori->n; \
- src_type_t max = src[0]; \
- for (l=1; l<fmt_ori->n; l++) if ( max < src[l] ) max = src[l]; \
+ src = fmt_ori->p + sizeof(src_type_t)*j*fmt_ori->n; \
+ src_type_t max = convert(src); \
+ for (l=1; l<fmt_ori->n; l++) \
+ { \
+ src_type_t val = convert(&src[l * sizeof(src_type_t)]); \
+ if ( max < val ) max = val; \
+ } \
for (l=0; l<nsize; l++) { *tgt = max; tgt++; } \
} \
else \
tgt = (tgt_type_t *) ma->tmp_arr + (ismpl+j)*nsize + inew; \
if ( src_is_vector_end ) break; \
if ( src_is_missing ) tgt_set_missing; \
- else *tgt = *src; \
- src++; \
+ else *tgt = convert(src); \
+ src += sizeof(src_type_t); \
} \
} \
} \
}
switch (type)
{
- case BCF_BT_INT8: BRANCH(int32_t, int8_t, *src==bcf_int8_missing, *src==bcf_int8_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
- case BCF_BT_INT16: BRANCH(int32_t, int16_t, *src==bcf_int16_missing, *src==bcf_int16_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
- case BCF_BT_INT32: BRANCH(int32_t, int32_t, *src==bcf_int32_missing, *src==bcf_int32_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
- case BCF_BT_FLOAT: BRANCH(float, float, bcf_float_is_missing(*src), bcf_float_is_vector_end(*src), bcf_float_set_missing(*tgt), bcf_float_set_vector_end(*tgt)); break;
+ case BCF_BT_INT8: BRANCH(int32_t, int8_t, le_to_i8, le_to_i8(src)==bcf_int8_missing, le_to_i8(src)==bcf_int8_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
+ case BCF_BT_INT16: BRANCH(int32_t, int16_t, le_to_i16, le_to_i16(src)==bcf_int16_missing, le_to_i16(src)==bcf_int16_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
+ case BCF_BT_INT32: BRANCH(int32_t, int32_t, le_to_i32, le_to_i32(src)==bcf_int32_missing, le_to_i32(src)==bcf_int32_vector_end, *tgt=bcf_int32_missing, *tgt=bcf_int32_vector_end); break;
+ case BCF_BT_FLOAT: BRANCH(float, float, le_to_float, bcf_float_is_missing(le_to_float(src)), bcf_float_is_vector_end(le_to_float(src)), bcf_float_set_missing(*tgt), bcf_float_set_vector_end(*tgt)); break;
default: error("Unexpected case: %d, %s\n", type, key);
}
#undef BRANCH
}
else
bcf_update_info_int32(args->out_hdr, out, "END", NULL, 0);
+
+ int iunseen;
+ if ( args->trim_star_allele && (out->n_allele > 2 || args->trim_star_allele > 1) && (iunseen=get_unseen_allele(out)) && iunseen>0 )
+ {
+ // the unobserved star allele should be trimmed, either it is variant site or trimming of all sites was requested
+ kbitset_t *rm_set = kbs_init(out->n_allele);
+ kbs_insert(rm_set, iunseen);
+ if ( bcf_remove_allele_set(args->out_hdr,out,rm_set) )
+ error("[%s] Error: failed to trim the unobserved allele at %s:%"PRIhts_pos"\n",__func__,bcf_seqname(args->out_hdr,out),out->pos+1);
+ kbs_destroy(rm_set);
+ }
if ( bcf_write1(args->out_fh, args->out_hdr, out)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
bcf_clear1(out);
-
// Inactivate blocks which do not extend beyond END and find new gvcf_min
min = INT_MAX;
for (i=0; i<args->files->nreaders; i++)
if ( args->do_gvcf )
bcf_update_info_int32(args->out_hdr, out, "END", NULL, 0);
merge_format(args, out);
+ int iunseen;
+ if ( args->trim_star_allele && (out->n_allele > 2 || args->trim_star_allele > 1) && (iunseen=get_unseen_allele(out)) && iunseen>0 )
+ {
+ // the unobserved star allele should be trimmed, either it is variant site or trimming of all sites was requested
+ kbitset_t *rm_set = kbs_init(out->n_allele);
+ kbs_insert(rm_set, iunseen);
+ if ( bcf_remove_allele_set(args->out_hdr,out,rm_set) )
+ error("[%s] Error: failed to trim the unobserved allele at %s:%"PRIhts_pos"\n",__func__,bcf_seqname(args->out_hdr,out),out->pos+1);
+ kbs_destroy(rm_set);
+ }
if ( bcf_write1(args->out_fh, args->out_hdr, out)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
bcf_clear1(out);
}
if ( hts_close(args->out_fh)!=0 ) error("[%s] Error: close failed .. %s\n", __func__,args->output_fname);
return;
}
- else if ( args->write_index && init_index(args->out_fh,args->out_hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ else if ( init_index2(args->out_fh,args->out_hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
- if ( args->collapse==COLLAPSE_NONE ) args->vcmp = vcmp_init();
+ args->vcmp = vcmp_init();
args->maux = maux_init(args);
args->out_line = bcf_init1();
args->tmph = kh_init(strdict);
fprintf(bcftools_stderr, "Usage: bcftools merge [options] <A.vcf.gz> <B.vcf.gz> [...]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Options:\n");
+ fprintf(bcftools_stderr, " --force-no-index Merge unindexed files, synonymous to --no-index\n");
fprintf(bcftools_stderr, " --force-samples Resolve duplicate sample names\n");
+ fprintf(bcftools_stderr, " --force-single Run even if there is only one file on input\n");
fprintf(bcftools_stderr, " --print-header Print only the merged header and exit\n");
fprintf(bcftools_stderr, " --use-header FILE Use the provided header\n");
fprintf(bcftools_stderr, " -0 --missing-to-ref Assume genotypes at missing sites are 0/0\n");
fprintf(bcftools_stderr, " -f, --apply-filters LIST Require at least one of the listed FILTER strings (e.g. \"PASS,.\")\n");
fprintf(bcftools_stderr, " -F, --filter-logic x|+ Remove filters if some input is PASS (\"x\"), or apply all filters (\"+\") [+]\n");
- fprintf(bcftools_stderr, " -g, --gvcf -|REF.FA Merge gVCF blocks, INFO/END tag is expected. Implies -i QS:sum,MinDP:min,I16:sum,IDV:max,IMF:max -M PL:max,AD:0\n");
+ fprintf(bcftools_stderr, " -g, --gvcf -|REF.FA Merge gVCF blocks, INFO/END tag is expected. Implies -i QS:sum,MinDP:min,MIN_DP:min,I16:sum,IDV:max,IMF:max -M PL:max,AD:0\n");
fprintf(bcftools_stderr, " -i, --info-rules TAG:METHOD,.. Rules for merging INFO fields (method is one of sum,avg,min,max,join) or \"-\" to turn off the default [DP:sum,DP4:sum]\n");
fprintf(bcftools_stderr, " -l, --file-list FILE Read file names from the file\n");
- fprintf(bcftools_stderr, " -L, --local-alleles INT EXPERIMENTAL: if more than <int> ALT alleles are encountered, drop FMT/PL and output LAA+LPL instead; 0=unlimited [0]\n");
- fprintf(bcftools_stderr, " -m, --merge STRING Allow multiallelic records for <snps|indels|both|snp-ins-del|all|none|id>, see man page for details [both]\n");
+ fprintf(bcftools_stderr, " -L, --local-alleles INT If more than INT alt alleles are encountered, drop FMT/PL and output LAA+LPL instead; 0=unlimited [0]\n");
+ fprintf(bcftools_stderr, " -m, --merge STRING[*|**] Allow multiallelic records for snps,indels,both,snp-ins-del,all,none,id,*,**; see man page for details [both]\n");
fprintf(bcftools_stderr, " -M, --missing-rules TAG:METHOD Rules for replacing missing values in numeric vectors (.,0,max) when unknown allele <*> is not present [.]\n");
fprintf(bcftools_stderr, " --no-index Merge unindexed files, the same chromosomal order is required and -r/-R are not allowed\n");
fprintf(bcftools_stderr, " --no-version Do not append version and command line to the header\n");
fprintf(bcftools_stderr, " -r, --regions REGION Restrict to comma-separated list of regions\n");
fprintf(bcftools_stderr, " -R, --regions-file FILE Restrict to regions listed in a file\n");
fprintf(bcftools_stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
- fprintf(bcftools_stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
{"missing-rules",required_argument,NULL,'M'},
{"no-version",no_argument,NULL,8},
{"no-index",no_argument,NULL,10},
+ {"force-no-index",no_argument,NULL,10},
+ {"force-single",no_argument,NULL,12},
{"filter-logic",required_argument,NULL,'F'},
- {"write-index",no_argument,NULL,11},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hm:f:r:R:o:O:i:M:l:g:F:0L:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hm:f:r:R:o:O:i:M:l:g:F:0L:W::",loptions,NULL)) >= 0) {
switch (c) {
case 'L':
args->local_alleles = strtol(optarg,&tmp,10);
}
break;
case 'm':
+ {
+ int len = strlen(optarg);
+ if ( optarg[len-1]=='*' ) { args->trim_star_allele++; len--; }
+ if ( optarg[len-1]=='*' ) { args->trim_star_allele++; len--; }
+ if ( optarg[len-1]==',' ) len--;
args->collapse = COLLAPSE_NONE;
- if ( !strcmp(optarg,"snps") ) args->collapse |= COLLAPSE_SNPS;
- else if ( !strcmp(optarg,"indels") ) args->collapse |= COLLAPSE_INDELS;
- else if ( !strcmp(optarg,"both") ) args->collapse |= COLLAPSE_BOTH;
- else if ( !strcmp(optarg,"any") ) args->collapse |= COLLAPSE_ANY;
- else if ( !strcmp(optarg,"all") ) args->collapse |= COLLAPSE_ANY;
- else if ( !strcmp(optarg,"none") ) args->collapse = COLLAPSE_NONE;
- else if ( !strcmp(optarg,"snp-ins-del") ) args->collapse = COLLAPSE_SNP_INS_DEL|COLLAPSE_SNPS;
- else if ( !strcmp(optarg,"id") ) { args->collapse = COLLAPSE_NONE; args->merge_by_id = 1; }
+ if ( !strncmp(optarg,"snp-ins-del",len) ) args->collapse = COLLAPSE_SNP_INS_DEL|COLLAPSE_SNPS;
+ else if ( !strncmp(optarg,"snps",len) ) args->collapse |= COLLAPSE_SNPS;
+ else if ( !strncmp(optarg,"indels",len) ) args->collapse |= COLLAPSE_INDELS;
+ else if ( !strncmp(optarg,"id",len) ) { args->collapse = COLLAPSE_NONE; args->merge_by_id = 1; }
+ else if ( !strncmp(optarg,"any",len) ) args->collapse |= COLLAPSE_ANY;
+ else if ( !strncmp(optarg,"all",len) ) args->collapse |= COLLAPSE_ANY;
+ else if ( !strncmp(optarg,"both",len) ) args->collapse |= COLLAPSE_BOTH;
+ else if ( !strncmp(optarg,"none",len) ) args->collapse = COLLAPSE_NONE;
else error("The -m type \"%s\" is not recognised.\n", optarg);
break;
+ }
case 'f': args->files->apply_filters = optarg; break;
case 'r': args->regions_list = optarg; break;
case 'R': args->regions_list = optarg; regions_is_file = 1; break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
case 10 : args->no_index = 1; break;
- case 11 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
+ case 12 : args->force_single = 1; break;
case 'h':
case '?': usage(); break;
default: error("Unknown argument: %s\n", optarg);
}
}
if ( argc==optind && !args->file_list ) usage();
- if ( argc-optind<2 && !args->file_list ) usage();
-
if ( args->no_index )
{
if ( args->regions_list ) error("Error: cannot combine --no-index with -r/-R\n");
for (i=0; i<nfiles; i++) free(files[i]);
free(files);
}
+ if ( !args->files->nreaders ) usage();
+ if ( args->files->nreaders==1 && !args->force_single ) error("Expected two or more files to merge, got only one. Use --force-single to proceed anyway\n");
+
merge_vcf(args);
bcf_sr_destroy(args->files);
if ( args->regs ) regidx_destroy(args->regs);
/* vcfnorm.c -- Left-align and normalize indels.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include "abuf.h"
#include "gff.h"
#include "regidx.h"
+#include "filter.h"
#define CHECK_REF_EXIT 1
#define CHECK_REF_WARN 2
#define MROWS_SPLIT 1
#define MROWS_MERGE 2
+// Logic of the filters: include or exclude sites which match the filters?
+#define FLT_INCLUDE 1
+#define FLT_EXCLUDE 2
+
// for -m+, mapping from allele indexes of a single input record
// to allele indexes of output record
typedef struct
{
int n; // number of alleles
char *ref, *alt;
- void *hash;
+ void *hash; // str2int hash
}
cmpals1_t;
{
char *tseq, *seq;
int mseq;
- bcf1_t **lines, **tmp_lines, **alines, **blines, *mrow_out;
- int ntmp_lines, mtmp_lines, nalines, malines, nblines, mblines;
+ bcf1_t **lines, **tmp_lines, **mrows, *mrow_out;
+ int ntmp_lines, mtmp_lines, nmrows, mmrows, mrows_first;
map_t *maps; // mrow map for each buffered record
char **als;
int mmaps, nals, mals;
int32_t *int32_arr;
int ntmp_arr1, ntmp_arr2, nint32_arr;
kstring_t *tmp_str;
- kstring_t *tmp_als, *tmp_del, tmp_kstr;
- int ntmp_als, ntmp_del;
+ kstring_t *tmp_als, *tmp_sym, tmp_kstr;
+ int ntmp_als, ntmp_sym;
rbuf_t rbuf;
int buf_win; // maximum distance between two records to consider
int aln_win; // the realignment window size (maximum repeat size)
struct { int tot, set, swap; } nref;
char **argv, *output_fname, *ref_fname, *vcf_fname, *region, *targets;
int argc, rmdup, output_type, n_threads, check_ref, strict_filter, do_indels, clevel;
- int nchanged, nskipped, nsplit, ntotal, mrows_op, mrows_collapse, parsimonious;
+ int nchanged, nskipped, nsplit, njoined, ntotal, nfilter, mrows_op, mrows_collapse, parsimonious;
int record_cmd_line, force, force_warned, keep_sum_ad;
abuf_t *abuf;
abuf_opt_t atomize;
char *old_rec_tag;
htsFile *out;
char *index_fn;
- int write_index;
+ int write_index, gff_verbosity;
int right_align;
char *gff_fname;
gff_t *gff;
regidx_t *idx_tscript;
regitr_t *itr_tscript;
+ int (*cmp_func)(const void *aptr, const void *bptr);
+ char *filter_str;
+ int filter_logic; // include or exclude sites which match the filters? One of FLT_INCLUDE/FLT_EXCLUDE
+ int filter_pass;
+ filter_t *filter;
}
args_t;
if ( bcf_get_variant_types(line)==VCF_BND ) return ERR_SYMBOLIC; // breakend, not an error
// make a copy of each allele for trimming
- hts_expand0(kstring_t,line->n_allele,args->ntmp_als,args->tmp_als);
- hts_expand0(kstring_t,line->n_allele,args->ntmp_del,args->tmp_del);
+ hts_expand0(kstring_t,line->n_allele,args->ntmp_als,args->tmp_als); // the actual sequence to realign
+ hts_expand0(kstring_t,line->n_allele,args->ntmp_sym,args->tmp_sym); // the original symbolic allele strings to output
kstring_t *als = args->tmp_als;
- kstring_t *del = args->tmp_del;
+ kstring_t *sym = args->tmp_sym;
+ int symbolic_alts = 1;
for (i=0; i<line->n_allele; i++)
{
- del[i].l = 0;
+ sym[i].l = 0;
if ( line->d.allele[i][0]=='<' )
{
- // symbolic allele, only <DEL.*> will be realigned
- if ( strncmp("<DEL",line->d.allele[i],4) ) return ERR_SYMBOLIC;
- if ( nref < line->rlen )
+ // symbolic allele, only <DEL.*> and <DUP.*> will be realigned
+ // TODO: there should be check for symbolic allele length. If too big, perhaps should not attempt realignment
+ int32_t sv_len = 0;
+ if ( !strncmp("<DEL",line->d.allele[i],4) ) sv_len = -line->rlen;
+ else if ( !strncmp("<DUP",line->d.allele[i],4) )
+ {
+ if ( bcf_get_info_int32(args->hdr,line,"SVLEN",&args->int32_arr,&args->nint32_arr)==1 ) sv_len = args->int32_arr[0];
+ }
+ if ( !sv_len ) return ERR_SYMBOLIC;
+
+ als[i].l = 0;
+ if ( sv_len<0 )
+ {
+ // del, expand REF and replace ALT, for example, replace "REF=C ALT=<DEL>" with "REF=CAT ALT=C"
+ if ( nref < line->rlen )
+ {
+ free(ref);
+ reflen = line->rlen;
+ ref = faidx_fetch_seq(args->fai, (char*)args->hdr->id[BCF_DT_CTG][line->rid].key, line->pos, line->pos+reflen-1, &nref);
+ if ( !ref ) error("faidx_fetch_seq failed at %s:%"PRId64"\n", args->hdr->id[BCF_DT_CTG][line->rid].key, (int64_t) line->pos+1);
+ seq_to_upper(ref,0);
+ replace_iupac_codes(ref,nref); // any non-ACGT character in fasta ref is replaced with N
+ als[0].l = 0;
+ kputs(ref, &als[0]);
+ }
+ kputsn(als[0].s,1,&als[i]);
+ }
+ else // sv_len>0
{
+ // dup, replace "REF=C ALT=<DUP>" with "REF=C ALT=CAT"
free(ref);
- reflen = line->rlen;
- ref = faidx_fetch_seq(args->fai, (char*)args->hdr->id[BCF_DT_CTG][line->rid].key, line->pos, line->pos+reflen-1, &nref);
+ ref = faidx_fetch_seq(args->fai, (char*)args->hdr->id[BCF_DT_CTG][line->rid].key, line->pos, line->pos+sv_len, &nref);
if ( !ref ) error("faidx_fetch_seq failed at %s:%"PRId64"\n", args->hdr->id[BCF_DT_CTG][line->rid].key, (int64_t) line->pos+1);
seq_to_upper(ref,0);
replace_iupac_codes(ref,nref); // any non-ACGT character in fasta ref is replaced with N
- als[0].l = 0;
- kputs(ref, &als[0]);
- als[i].l = 0;
- kputsn(ref,1,&als[i]);
- kputs(line->d.allele[i],&del[i]);
- continue;
+ kputs(ref,&als[i]);
}
+ kputs(line->d.allele[i],&sym[i]); // preserve the symbolic allele string
+ continue;
}
+ if ( i>0 ) symbolic_alts = 0;
if ( line->d.allele[i][0]=='*' ) return ERR_SPANNING_DELETION; // spanning deletion
if ( has_non_acgtn(line->d.allele[i],line->shared.l) )
{
else
new_pos = realign_right(args, line);
- // Have the alleles changed?
- als[0].s[ als[0].l ] = 0; // in order for strcmp to work
+ // Have the alleles changed? Consider <DEL> could have expanded the REF allele. In that
+ // case it must be trimmed, however the new REF length must reflect the entire length.
+ als[0].s[ als[0].l ] = 0; // for strcmp to work
+ int new_reflen = strlen(als[0].s);
+ if ( symbolic_alts )
+ {
+ als[0].l = 1;
+ als[0].s[ als[0].l ] = 0;
+ }
if ( new_pos==line->pos && !strcasecmp(line->d.allele[0],als[0].s) ) return ERR_OK;
set_old_rec_tag(args, line, line, 0);
for (i=0; i<line->n_allele; i++)
{
if (i>0) kputc(',',&args->tmp_kstr);
- if ( del[i].l ) kputs(del[i].s,&args->tmp_kstr);
+ if ( sym[i].l ) kputs(sym[i].s,&args->tmp_kstr);
else kputsn(als[i].s,als[i].l,&args->tmp_kstr);
}
args->tmp_kstr.s[ args->tmp_kstr.l ] = 0;
args->nchanged++;
// Update INFO/END if necessary
- int new_reflen = strlen(line->d.allele[0]);
if ( (new_pos!=line->pos || reflen!=new_reflen) && bcf_get_info_int32(args->hdr, line, "END", &args->int32_arr, &args->nint32_arr)==1 )
{
// bcf_update_alleles_str() messed up rlen because line->pos changed. This will be fixed by bcf_update_info_int32()
bcf_update_info_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele-1,ret); \
} \
bcf_update_info_##type(args->out_hdr,dst,tag,vals+ialt,1); \
bcf_update_info_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele,ret); \
} \
if ( args->keep_sum_ad >= 0 && args->keep_sum_ad==info->key ) \
bcf_update_info_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele*(src->n_allele+1)/2,ret); \
} \
if ( ialt!=0 ) \
}
#undef BRANCH_NUMERIC
}
-// Find n-th field in a comma-separated list and move it to dst.
-// The memory areas may overlap.
-#define STR_MOVE_NTH(dst,src,end,nth,len) \
-{ \
- char *ss = src, *se = src; \
- int j = 0; \
- while ( *se && se<(end) ) \
- { \
- if ( *se==',' ) \
- { \
- if ( j==nth ) break; \
- j++; \
- ss = se+1; \
- } \
- se++; \
- } \
- if ( j==nth ) \
- { \
- int n = se - ss; \
- memmove((dst),ss,n); \
- src = se; \
- len += n; \
- } \
- else len = -1; \
+// Find nth field in a comma-separated list in src and move it to dst.
+// The dst and src memory areas may overlap, end points just after the last valid src
+// character.
+// On success returns pointer to the end of the parsed field and increments ndst by the
+// number of memmoved characters. Returns NULL if the field was not found.
+static inline char *string_move_nth(char *dst, char *src, char *end, int nth, size_t *ndst)
+{
+ if ( src>=end ) return NULL;
+ char *ss = src, *se = src;
+ int j = 0;
+ while ( *se && se<(end) )
+ {
+ if ( *se==',' )
+ {
+ if ( j==nth ) break;
+ j++;
+ ss = se+1;
+ }
+ se++;
+ }
+ if ( j!=nth ) return NULL;
+ if ( ss>=end ) return NULL;
+ if ( !*ss ) return NULL;
+
+ int n = se - ss;
+ memmove((dst),ss,n);
+ *ndst += n;
+ return se;
}
+
static void split_info_string(args_t *args, bcf1_t *src, bcf_info_t *info, int ialt, bcf1_t *dst)
{
const char *tag = bcf_hdr_int2id(args->hdr,BCF_DT_ID,info->key);
int len = bcf_hdr_id2length(args->hdr,BCF_HL_INFO,info->key);
if ( len==BCF_VL_A )
{
+ char *end = str.s + str.l;
char *tmp = str.s;
- int len = 0;
- STR_MOVE_NTH(str.s,tmp,str.s+str.l,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len] = 0;
+ str.l = 0;
+ tmp = string_move_nth(str.s,tmp,end,ialt,&str.l);
+ if ( !tmp ) str.l = 1, str.s[0] = '.';
+ kputc_(0,&str);
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
}
else if ( len==BCF_VL_R )
{
+ char *end = str.s + str.l;
char *tmp = str.s;
- int len = 0;
- STR_MOVE_NTH(str.s,tmp,str.s+str.l,0,len);
- str.s[len]=','; tmp++; len++;
- STR_MOVE_NTH(&str.s[len],tmp,str.s+str.l,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len] = 0;
+ str.l = 0;
+ tmp = string_move_nth(str.s,tmp,end,0,&str.l);
+ if ( tmp )
+ {
+ kputc_(',',&str);
+ tmp = string_move_nth(str.s+str.l,tmp+1,end,ialt,&str.l); // ialt is 0-based index to ALT
+ }
+ if ( !tmp ) str.l = 1, str.s[0] = '.';
+ kputc_(0,&str);
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
}
else if ( len==BCF_VL_G )
{
int i0a = bcf_alleles2gt(0,ialt+1), iaa = bcf_alleles2gt(ialt+1,ialt+1);
+ char *end = str.s + str.l;
char *tmp = str.s;
- int len = 0;
- STR_MOVE_NTH(str.s,tmp,str.s+str.l,0,len);
- str.s[len]=','; tmp++; len++;
- STR_MOVE_NTH(&str.s[len],tmp,str.s+str.l,i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len]=','; tmp++; len++;
- STR_MOVE_NTH(&str.s[len],tmp,str.s+str.l,iaa-i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len] = 0;
+ str.l = 0;
+ tmp = string_move_nth(str.s,tmp,end,0,&str.l);
+ if ( tmp )
+ {
+ kputc_(',',&str);
+ tmp = string_move_nth(str.s+str.l,tmp+1,end,i0a-1,&str.l);
+ }
+ if ( tmp )
+ {
+ kputc_(',',&str);
+ tmp = string_move_nth(str.s+str.l,tmp+1,end,iaa-i0a-1,&str.l);
+ }
+ if ( !tmp ) str.l = 1, str.s[0] = '.';
+ kputc_(0,&str);
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
}
else
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
+ if ( args->ntmp_arr1 < str.m )
+ {
+ args->ntmp_arr1 = str.m;
+ args->tmp_arr1 = (uint8_t*)str.s;
+ }
}
static void split_info_flag(args_t *args, bcf1_t *src, bcf_info_t *info, int ialt, bcf1_t *dst)
{
{
if ( gt[j]==bcf_int32_vector_end ) break;
if ( bcf_gt_is_missing(gt[j]) ) continue; // missing allele: leave as is
- if ( (ialt==0 || args->ma_use_ref_allele) && bcf_gt_allele(gt[j])==0 ) continue; // ref && `--multi-overlaps 0`: leave as is
+ if ( bcf_gt_allele(gt[j])==0 ) continue; // ref && `--multi-overlaps 0`: leave as is
if ( bcf_gt_allele(gt[j])==ialt+1 )
gt[j] = bcf_gt_unphased(1) | bcf_gt_is_phased(gt[j]); // set to first ALT
else if ( args->ma_use_ref_allele )
bcf_update_format_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,(src->n_allele-1)*nsmpl,nvals); \
} \
nvals /= nsmpl; \
bcf_update_format_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele*nsmpl,nvals); \
} \
nvals /= nsmpl; \
bcf_update_format_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error at %s:%"PRId64", the tag %s has wrong number of fields\n", bcf_seqname(args->hdr,src),(int64_t) src->pos+1,bcf_hdr_int2id(args->hdr,BCF_DT_ID,fmt->id)); \
+ error("Error at %s:%"PRId64", the tag %s has wrong number of fields. Use --force to proceed anyway.\n", \
+ bcf_seqname(args->hdr,src),(int64_t) src->pos+1,bcf_hdr_int2id(args->hdr,BCF_DT_ID,fmt->id)); \
} \
nvals /= nsmpl; \
int all_haploid = nvals==src->n_allele ? 1 : 0; \
isrc += src_blen;
}
}
+// ialt is 0-based index to ALT
static void split_format_string(args_t *args, bcf1_t *src, bcf_fmt_t *fmt, int ialt, bcf1_t *dst)
{
const char *tag = bcf_hdr_int2id(args->hdr,BCF_DT_ID,fmt->id);
if ( !ret ) return; // all values can be empty, leave out the tag, no need to panic
assert( ret>0 );
+ int nsmpl = bcf_hdr_nsamples(args->hdr);
+ int blen = ret/nsmpl; // per-sample field length
+ assert( blen>0 );
+
kstring_t str;
str.m = args->ntmp_arr1;
- str.l = ret;
str.s = (char*) args->tmp_arr1;
+ str.l = ret;
- int nsmpl = bcf_hdr_nsamples(args->hdr);
- int len = bcf_hdr_id2length(args->hdr,BCF_HL_FMT,fmt->id);
- if ( len==BCF_VL_A )
+ int tag_len = bcf_hdr_id2length(args->hdr,BCF_HL_FMT,fmt->id);
+ if ( tag_len==BCF_VL_A )
{
- int i, blen = ret/nsmpl, maxlen = 0;
+ int i, maxlen = 0;
char *ptr = str.s;
for (i=0; i<nsmpl; i++)
{
char *tmp = ptr;
- int len = 0;
- STR_MOVE_NTH(tmp,tmp,ptr+blen,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ char *end = ptr + blen;
+ size_t len = 0;
+ tmp = string_move_nth(ptr,tmp,end,ialt,&len);
+ if ( !tmp ) ptr[0] = '.', len = 1;
if ( maxlen < len ) maxlen = len;
+ while (len<blen) ptr[len++] = 0;
ptr += blen;
}
if ( maxlen<blen ) squeeze_format_char(str.s,blen,maxlen,nsmpl);
bcf_update_format_char(args->out_hdr,dst,tag,str.s,nsmpl*maxlen);
}
- else if ( len==BCF_VL_R )
+ else if ( tag_len==BCF_VL_R )
{
- int i, blen = ret/nsmpl, maxlen = 0;
+ int i, maxlen = 0;
char *ptr = str.s;
for (i=0; i<nsmpl; i++)
{
char *tmp = ptr;
- int len = 0;
- STR_MOVE_NTH(ptr,tmp,ptr+blen,0,len);
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ char *end = ptr + blen;
+ size_t len = 0;
+ tmp = string_move_nth(ptr,tmp,end,0,&len);
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,ialt,&len);
+ }
+ if ( !tmp ) ptr[0] = '.', len = 1;
if ( maxlen < len ) maxlen = len;
+ while (len<blen) ptr[len++] = 0;
ptr += blen;
}
if ( maxlen<blen ) squeeze_format_char(str.s,blen,maxlen,nsmpl);
bcf_update_format_char(args->out_hdr,dst,tag,str.s,nsmpl*maxlen);
}
- else if ( len==BCF_VL_G )
+ else if ( tag_len==BCF_VL_G )
{
- int i, blen = ret/nsmpl, maxlen = 0, i0a = bcf_alleles2gt(0,ialt+1), iaa = bcf_alleles2gt(ialt+1,ialt+1);
+ int i, maxlen = 0, i0a = bcf_alleles2gt(0,ialt+1), iaa = bcf_alleles2gt(ialt+1,ialt+1);
char *ptr = str.s;
for (i=0; i<nsmpl; i++)
{
bcf_update_format_char(args->out_hdr,dst,tag,NULL,0);
return;
}
- error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d or %d, found %d\n",
+ error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d or %d, found %d. Use --force to proceed anyway.\n",
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele*(src->n_allele+1)/2,src->n_allele,nfields);
}
- int len = 0;
+ char *tmp = ptr;
+ char *end = ptr + blen;
+ size_t len = 0;
+ tmp = string_move_nth(ptr,tmp,end,0,&len);
if ( nfields==src->n_allele ) // haploid
{
- char *tmp = ptr;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,0,len);
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,ialt,&len);
+ }
}
else // diploid
{
- char *tmp = ptr;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,0,len);
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,iaa-i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,i0a-1,&len);
+ }
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,iaa-i0a-1,&len);
+ }
}
+ if ( !tmp ) ptr[0] = '.', len = 1;
if ( maxlen < len ) maxlen = len;
+ while (len<blen) ptr[len++] = 0;
ptr += blen;
}
if ( maxlen<blen ) squeeze_format_char(str.s,blen,maxlen,nsmpl);
bcf_update_info_string(args->out_hdr,dst,tag,args->tmp_arr1);
}
}
+static int gt_array_grow_ploidy(args_t *args, uint8_t **tmp_arr, int *ntmp_arr, int ngt_ori, int ngt_new, int nsmpl)
+{
+ *ntmp_arr = 4*ngt_new*nsmpl;
+ int32_t *ptr = (int32_t*)realloc(*tmp_arr,*ntmp_arr);
+ if ( !ptr ) error("Error: failed to allocate %d bytes\n",*ntmp_arr);
+ *tmp_arr = (uint8_t*) ptr;
+
+ int i,j;
+ for (i=nsmpl-1; i>=0; i--)
+ {
+ int32_t *src = ptr + i*ngt_ori;
+ int32_t *dst = ptr + i*ngt_new;
+ for (j=ngt_new; j>ngt_ori; j--) dst[j-1] = bcf_int32_vector_end;
+ for (j=ngt_ori; j>0; j--) dst[j-1] = src[j-1];
+ }
+ return ngt_new;
+}
static void merge_format_genotype(args_t *args, bcf1_t **lines, int nlines, bcf_fmt_t *fmt, bcf1_t *dst)
{
// reusing int8_t arrays as int32_t arrays
int ngts2 = bcf_get_genotypes(args->hdr,lines[i],&args->tmp_arr2,&ntmp2);
args->ntmp_arr2 = ntmp2 * 4;
ngts2 /= nsmpl;
- if ( ngts!=ngts2 ) error("Error at %s:%"PRId64": cannot combine diploid with haploid genotype\n", bcf_seqname(args->hdr,lines[i]),(int64_t) lines[i]->pos+1);
+ int ploidy_changed = ngts - ngts2;
+ if ( ngts < ngts2 ) ngts = gt_array_grow_ploidy(args,&args->tmp_arr1,&args->ntmp_arr1,ngts,ngts2,nsmpl);
+ if ( ngts > ngts2 ) ngts2 = gt_array_grow_ploidy(args,&args->tmp_arr2,&args->ntmp_arr2,ngts2,ngts,nsmpl);
int32_t *gt = (int32_t*) args->tmp_arr1; // the first, destination line
int32_t *gt2 = (int32_t*) args->tmp_arr2; // one of the subsequent lines, i.e. the source line
// never overwrite with ref allele
for (k2=0; k2<ngts2; k2++)
{
- if ( gt2[k2]==bcf_int32_vector_end ) break;
- if ( bcf_gt_is_missing(gt2[k2]) ) continue;
+ if ( gt2[k2]==bcf_int32_vector_end )
+ {
+ if ( ploidy_changed && bcf_gt_is_missing(gt[k2]) ) gt[k2] = bcf_int32_vector_end;
+ break;
+ }
+ if ( bcf_gt_is_missing(gt2[k2]) ) continue; // don't overwrite with missing
+
+ // don't overwrite with ref, unless the destination is missing, e.g. "./. + 0/1"
int ial2 = bcf_gt_allele(gt2[k2]);
- if ( ial2==0 ) continue; // never overwrite with ref
+ if ( ial2==0 && !bcf_gt_is_missing(gt[k2]) && gt[k2]!=bcf_int32_vector_end ) continue;
if ( ial2>=args->maps[i].nals ) error("Error at %s:%"PRId64": incorrect allele index %d\n",bcf_seqname(args->hdr,lines[i]),(int64_t) lines[i]->pos+1,ial2);
// The destination allele
int ial = args->maps[i].map[ial2];
if ( gt[k2]==bcf_int32_vector_end || bcf_gt_is_missing(gt[k2]) || !bcf_gt_allele(gt[k2]) )
- gt[k2] = bcf_gt_is_phased(gt[k2]) ? bcf_gt_phased(ial) : bcf_gt_unphased(ial);
+ gt[k2] = (gt[k2]!=bcf_int32_vector_end && bcf_gt_is_phased(gt[k2])) ? bcf_gt_phased(ial) : bcf_gt_unphased(ial);
else
{
// conflict, the first line has non-zero allele, use the old way, possibly disrupt the phasing
else if ( type==BCF_HT_INT || type==BCF_HT_REAL ) merge_format_numeric(args, lines, nlines, fmt, dst);
else merge_format_string(args, lines, nlines, fmt, dst);
}
+ args->njoined++;
}
#define SWAP(type_t, a, b) { type_t t = a; a = b; b = t; }
-static void mrows_schedule(args_t *args, bcf1_t **line)
+static void mrows_push(args_t *args, bcf1_t **line)
{
int i,m;
- if ( args->mrows_collapse==COLLAPSE_ANY // merge all record types together
- || bcf_get_variant_types(*line)&VCF_SNP // SNP, put into alines
- || bcf_get_variant_types(*line)==VCF_REF ) // ref
- {
- args->nalines++;
- m = args->malines;
- hts_expand(bcf1_t*,args->nalines,args->malines,args->alines);
- for (i=m; i<args->malines; i++) args->alines[i] = bcf_init1();
- SWAP(bcf1_t*, args->alines[args->nalines-1], *line);
- }
- else
- {
- args->nblines++;
- m = args->mblines;
- hts_expand(bcf1_t*,args->nblines,args->mblines,args->blines);
- for (i=m; i<args->mblines; i++) args->blines[i] = bcf_init1();
- SWAP(bcf1_t*, args->blines[args->nblines-1], *line);
+ if ( !args->nmrows ) args->mrows_first = 0;
+ args->nmrows++;
+ m = args->mmrows;
+ hts_expand(bcf1_t*,args->nmrows,args->mmrows,args->mrows);
+ for (i=m; i<args->mmrows; i++) args->mrows[i] = bcf_init1();
+ SWAP(bcf1_t*, args->mrows[args->nmrows-1], *line);
+
+ if ( args->mrows_collapse==COLLAPSE_ANY ) return;
+
+ // move the line up the sorted list so that the same variant types end up together
+ int cur_type = bcf_get_variant_types(args->mrows[args->nmrows-1]);
+ i = args->mrows_first + args->nmrows - 1;
+ while (i>0)
+ {
+ int prev_type = bcf_get_variant_types(args->mrows[i-1]);
+ if ( prev_type <= cur_type ) break;
+ bcf1_t *tmp = args->mrows[i-1];
+ args->mrows[i-1] = args->mrows[i];
+ args->mrows[i] = tmp;
+ i--;
}
}
-static int mrows_ready_to_flush(args_t *args, bcf1_t *line)
+static int mrows_can_flush(args_t *args, bcf1_t *line)
{
- if ( args->nalines && (args->alines[0]->rid!=line->rid || args->alines[0]->pos!=line->pos) ) return 1;
- if ( args->nblines && (args->blines[0]->rid!=line->rid || args->blines[0]->pos!=line->pos) ) return 1;
+ if ( !args->nmrows ) return 0;
+ int ibeg = args->mrows_first;
+ if ( args->mrows[ibeg]->rid != line->rid ) return 1;
+ if ( args->mrows[ibeg]->pos != line->pos ) return 1;
return 0;
}
static bcf1_t *mrows_flush(args_t *args)
{
- if ( args->nblines && args->nalines==1 && bcf_get_variant_types(args->alines[0])==VCF_REF )
+ if ( !args->nmrows ) return NULL;
+
+ int ibeg = args->mrows_first;
+
+ //fprintf(stderr,"flush: ibeg=%d n=%d\n",ibeg,args->nmrows);
+ //int i;
+ //for (i=ibeg; i<ibeg+args->nmrows; i++)
+ // fprintf(stderr,"\ti=%d type=%d %s %s\n",i,bcf_get_variant_types(args->mrows[i]),args->mrows[i]->d.allele[0],args->mrows[i]->d.allele[1]);
+
+ if ( args->nmrows==1 )
{
- // By default, REF lines are merged with SNPs if SNPs and indels are to be kept separately.
- // However, if there are indels only and a single REF line, merge it with indels.
- args->nblines++;
- int i,m = args->mblines;
- hts_expand(bcf1_t*,args->nblines,args->mblines,args->blines);
- for (i=m; i<args->mblines; i++) args->blines[i] = bcf_init1();
- SWAP(bcf1_t*, args->blines[args->nblines-1], args->alines[0]);
- args->nalines--;
+ args->nmrows = 0;
+ return args->mrows[ibeg];
}
- if ( args->nalines )
+
+ if ( args->mrows_collapse==COLLAPSE_ANY )
{
- if ( args->nalines==1 )
- {
- args->nalines = 0;
- return args->alines[0];
- }
+ // merge everything with anything
bcf_clear(args->mrow_out);
- merge_biallelics_to_multiallelic(args, args->mrow_out, args->alines, args->nalines);
- args->nalines = 0;
+ merge_biallelics_to_multiallelic(args, args->mrow_out, &args->mrows[ibeg], args->nmrows - ibeg);
+ args->nmrows = 0;
return args->mrow_out;
}
- else if ( args->nblines )
+
+ int j;
+ int types[] = { VCF_SNP, VCF_MNP, VCF_INDEL, VCF_OTHER, -1 }; // merge everything within the same category
+ if ( args->mrows_collapse==COLLAPSE_SNPS ) types[1] = -1; // merge SNPs only
+ else if ( args->mrows_collapse==COLLAPSE_INDELS ) types[0] = VCF_INDEL, types[1] = -1; // merge indels only
+ for (j=0; types[j]!=-1; j++)
{
- if ( args->nblines==1 )
+ int i, type = types[j]; // to keep the compiler happy
+ for (i=ibeg; i<ibeg+args->nmrows; i++)
{
- args->nblines = 0;
- return args->blines[0];
+ type = bcf_get_variant_types(args->mrows[i]);
+ if ( type!=types[j] && type!=VCF_REF ) break;
+ }
+ if ( i==ibeg+1 && type!=VCF_REF )
+ {
+ // just one line of this type, no merging, but multiple lines of different type follow
+ args->nmrows--;
+ args->mrows_first++;
+ return args->mrows[ibeg];
+ }
+ if ( i>ibeg )
+ {
+ // more than one line, merging is needed
+ int nflush = i - ibeg;
+ bcf_clear(args->mrow_out);
+ merge_biallelics_to_multiallelic(args, args->mrow_out, &args->mrows[ibeg], nflush);
+ args->nmrows -= nflush;
+ args->mrows_first += nflush;
+ return args->mrow_out;
}
- bcf_clear(args->mrow_out);
- merge_biallelics_to_multiallelic(args, args->mrow_out, args->blines, args->nblines);
- args->nblines = 0;
- return args->mrow_out;
}
- return NULL;
+ args->nmrows--;
+ args->mrows_first++;
+ return args->mrows[ibeg];
}
-static void cmpals_add(cmpals_t *ca, bcf1_t *rec)
+static char *strdup_alt_svlen(args_t *args, bcf1_t *rec, int ial)
+{
+ if ( rec->d.allele[ial][0]!='<' ) return strdup(rec->d.allele[ial]);
+
+ int ntmp = args->ntmp_arr1 / sizeof(int32_t);
+ int n = bcf_get_info_int32(args->hdr, rec, "SVLEN", &args->tmp_arr1, &ntmp);
+ args->ntmp_arr1 = ntmp * sizeof(int32_t);
+ int32_t *svlen = (int32_t *) args->tmp_arr1;
+ if ( n<=0 ) return strdup(rec->d.allele[ial]);
+
+ if ( n+1 != rec->n_allele )
+ {
+ // there should be as many SVLEN numbers as there are ALT alleles
+ static int warned = 0;
+ if ( !warned )
+ {
+ fprintf(stderr,"TODO: different number of ALT alleles and SVLEN fields %s:%"PRIhts_pos"\n",bcf_seqname(args->hdr,rec),rec->pos+1);
+ warned = 1;
+ }
+ }
+
+ kstring_t str = {0,0,0};
+ ksprintf(&str,"%s.%d",rec->d.allele[ial],svlen[ial-1]);
+ return str.s;
+}
+static void cmpals_add(args_t *args, cmpals_t *ca, bcf1_t *rec)
{
ca->ncmpals++;
hts_expand0(cmpals1_t, ca->ncmpals, ca->mcmpals, ca->cmpals);
free(cmpals->ref);
cmpals->ref = strdup(rec->d.allele[0]);
cmpals->n = rec->n_allele;
+
if ( rec->n_allele==2 )
{
free(cmpals->alt);
- cmpals->alt = strdup(rec->d.allele[1]);
+ cmpals->alt = strdup_alt_svlen(args,rec,1);
}
else
{
khash_str2int_inc(cmpals->hash, strdup(rec->d.allele[i]));
}
}
-static int cmpals_match(cmpals_t *ca, bcf1_t *rec)
+static int cmpals_match(args_t *args, cmpals_t *ca, bcf1_t *rec)
{
int i, j;
+ char *alt_svlen = rec->n_allele==2 ? strdup_alt_svlen(args,rec,1) : NULL;
for (i=0; i<ca->ncmpals; i++)
{
cmpals1_t *cmpals = ca->cmpals + i;
// the most frequent case
if ( rec->n_allele==2 )
{
- if ( strcasecmp(rec->d.allele[1], cmpals->alt) ) continue;
+ if ( strcasecmp(alt_svlen, cmpals->alt) ) continue;
+ free(alt_svlen);
return 1;
}
if ( j<rec->n_allele ) continue;
return 1;
}
+ free(alt_svlen);
return 0;
}
static void cmpals_reset(cmpals_t *ca) { ca->ncmpals = 0; }
k = rbuf_shift(&args->rbuf);
if ( args->mrows_op==MROWS_MERGE )
{
- if ( mrows_ready_to_flush(args, args->lines[k]) )
+ if ( mrows_can_flush(args, args->lines[k]) )
{
while ( (line=mrows_flush(args)) )
if ( bcf_write1(file, args->out_hdr, line)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
}
- int merge = 1;
- if ( args->mrows_collapse!=COLLAPSE_BOTH && args->mrows_collapse!=COLLAPSE_ANY )
- {
- if ( !(bcf_get_variant_types(args->lines[k]) & args->mrows_collapse) ) merge = 0;
- }
- if ( merge )
- {
- mrows_schedule(args, &args->lines[k]);
- continue;
- }
+ mrows_push(args, &args->lines[k]);
+ continue;
}
else if ( args->rmdup )
{
if ( args->rmdup & BCF_SR_PAIR_ANY ) continue; // rmdup by position only
if ( args->rmdup & BCF_SR_PAIR_SNPS && line_type&(VCF_SNP|VCF_MNP) && prev_type&(VCF_SNP|VCF_MNP) ) continue;
if ( args->rmdup & BCF_SR_PAIR_INDELS && line_type&(VCF_INDEL) && prev_type&(VCF_INDEL) ) continue;
- if ( args->rmdup & BCF_SR_PAIR_EXACT && cmpals_match(&args->cmpals_out, args->lines[k]) ) continue;
+ if ( args->rmdup & BCF_SR_PAIR_EXACT && cmpals_match(args, &args->cmpals_out, args->lines[k]) ) continue;
}
else
{
if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_reset(&args->cmpals_out);
}
prev_type |= line_type;
- if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_add(&args->cmpals_out, args->lines[k]);
+ if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_add(args,&args->cmpals_out, args->lines[k]);
}
if ( bcf_write1(file, args->out_hdr, args->lines[k])!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
}
else
args->keep_sum_ad = -1;
- args->out_hdr = bcf_hdr_dup(args->hdr);
+ args->out_hdr = args->hdr;
if ( args->old_rec_tag )
bcf_hdr_printf(args->out_hdr,"##INFO=<ID=%s,Number=1,Type=String,Description=\"Original variant. Format: CHR|POS|REF|ALT|USED_ALT_IDX\">",args->old_rec_tag);
args->abuf = abuf_init(args->hdr, SPLIT);
abuf_set_opt(args->abuf, bcf_hdr_t*, BCF_HDR, args->out_hdr);
if ( args->old_rec_tag )
+ {
abuf_set_opt(args->abuf, const char*, INFO_TAG, args->old_rec_tag);
+ if ( bcf_hdr_sync(args->out_hdr)!=0 ) error("bcf_hdr_sync failed\n");
+ }
abuf_set_opt(args->abuf, int, STAR_ALLELE, args->use_star_allele);
}
if ( args->gff_fname )
{
args->gff = gff_init(args->gff_fname);
- gff_set(args->gff,verbosity,1);
+ gff_set(args->gff,verbosity,args->gff_verbosity);
gff_set(args->gff,strip_chr_names,1);
gff_parse(args->gff);
args->idx_tscript = gff_get(args->gff,idx_tscript);
args->itr_tscript = regitr_init(NULL);
}
+ if ( args->filter_str )
+ args->filter = filter_init(args->hdr, args->filter_str);
+ args->filter_pass = 1;
+
+ args->out_hdr = bcf_hdr_dup(args->out_hdr);
}
static void destroy_data(args_t *args)
{
+ if ( args->filter ) filter_destroy(args->filter);
if ( args->gff )
{
gff_destroy(args->gff);
for (i=0; i<args->mtmp_lines; i++)
if ( args->tmp_lines[i] ) bcf_destroy1(args->tmp_lines[i]);
free(args->tmp_lines);
- for (i=0; i<args->malines; i++)
- bcf_destroy1(args->alines[i]);
- free(args->alines);
- for (i=0; i<args->mblines; i++)
- bcf_destroy1(args->blines[i]);
- free(args->blines);
+ for (i=0; i<args->mmrows; i++)
+ bcf_destroy1(args->mrows[i]);
+ free(args->mrows);
for (i=0; i<args->mmaps; i++)
free(args->maps[i].map);
for (i=0; i<args->ntmp_als; i++)
free(args->tmp_als[i].s);
- for (i=0; i<args->ntmp_del; i++)
- free(args->tmp_del[i].s);
+ for (i=0; i<args->ntmp_sym; i++)
+ free(args->tmp_sym[i].s);
free(args->tmp_als);
- free(args->tmp_del);
+ free(args->tmp_sym);
free(args->tmp_kstr.s);
if ( args->tmp_str )
{
{
if ( args->fai )
{
- if ( args->check_ref & CHECK_REF_FIX ) fix_ref(args, line);
+ if ( args->filter_pass && (args->check_ref & CHECK_REF_FIX) ) fix_ref(args, line);
if ( args->do_indels )
{
- int ret = realign(args, line);
+ int ret = args->filter_pass ? realign(args, line) : ERR_OK;
// exclude broken VCF lines
if ( ret==ERR_REF_MISMATCH && args->check_ref & CHECK_REF_SKIP )
}
}
- // insert into sorted buffer
- rbuf_expand0(&args->rbuf,bcf1_t*,args->rbuf.n+1,args->lines);
- int i,j;
- i = j = rbuf_append(&args->rbuf);
- if ( args->lines[i] ) bcf_destroy(args->lines[i]);
- args->lines[i] = bcf_dup(line);
- while ( rbuf_prev(&args->rbuf,&i) )
+ if ( args->filter_pass && args->atomize==SPLIT ) abuf_push(args->abuf,line);
+ while (1)
{
- if ( args->lines[i]->pos > args->lines[j]->pos ) SWAP(bcf1_t*, args->lines[i], args->lines[j]);
- j = i;
+ if ( args->filter_pass && args->atomize==SPLIT )
+ {
+ line = abuf_flush(args->abuf, 0);
+ if ( !line ) break;
+ }
+
+ // insert into sorted buffer
+ rbuf_expand0(&args->rbuf,bcf1_t*,args->rbuf.n+1,args->lines);
+ int i,j;
+ i = j = rbuf_append(&args->rbuf);
+ if ( args->lines[i] ) bcf_destroy(args->lines[i]);
+ args->lines[i] = bcf_dup(line);
+ while ( rbuf_prev(&args->rbuf,&i) )
+ {
+ if ( args->lines[i]->rid==args->lines[j]->rid )
+ {
+ bcf_unpack(args->lines[i], BCF_UN_STR);
+ bcf_unpack(args->lines[j], BCF_UN_STR);
+ if ( args->cmp_func(&args->lines[i], &args->lines[j]) > 0) SWAP(bcf1_t*, args->lines[i], args->lines[j]);
+ }
+ j = i;
+ }
+ if ( !args->filter_pass || args->atomize!=SPLIT ) break;
}
}
-static bcf1_t *next_atomized_line(args_t *args)
+// return 0 on success, 1 when done
+static int split_and_normalize(args_t *args)
{
- bcf1_t *rec = NULL;
- if ( args->atomize==SPLIT )
+ if ( !bcf_sr_next_line(args->files) ) return 1;
+
+ bcf1_t *line = bcf_sr_get_line(args->files,0);
+ args->ntotal++;
+
+ if ( args->filter )
{
- rec = abuf_flush(args->abuf, 0);
- if ( rec ) return rec;
+ args->filter_pass = filter_test(args->filter,line,NULL);
+ if ( args->filter_logic==FLT_EXCLUDE ) args->filter_pass = args->filter_pass ? 0 : 1;
+ if ( !args->filter_pass ) args->nfilter++;
}
- if ( !bcf_sr_next_line(args->files) ) return NULL;
+ if ( args->mrows_op!=MROWS_SPLIT || line->n_allele<=2 || !args->filter_pass )
+ {
+ // normal operation, no splitting
+ normalize_line(args, line);
+ return 0;
+ }
- if ( args->atomize==SPLIT )
+ // any restrictions on variant types to split?
+ if ( args->mrows_collapse!=COLLAPSE_BOTH && args->mrows_collapse!=COLLAPSE_ANY )
{
- abuf_push(args->abuf,bcf_sr_get_line(args->files,0));
- return abuf_flush(args->abuf, 0);
+ int type = args->mrows_collapse==COLLAPSE_SNPS ? VCF_SNP : VCF_INDEL;
+ if ( !(bcf_get_variant_types(line) & type) )
+ {
+ normalize_line(args, line);
+ return 0;
+ }
}
- return bcf_sr_get_line(args->files,0);
+
+ args->nsplit++;
+ split_multiallelic_to_biallelics(args, line);
+
+ int j;
+ for (j=0; j<args->ntmp_lines; j++)
+ normalize_line(args, args->tmp_lines[j]);
+
+ return 0;
}
+
static void normalize_vcf(args_t *args)
{
char wmode[8];
hts_set_opt(args->out, HTS_OPT_THREAD_POOL, args->files->p);
if (args->record_cmd_line) bcf_hdr_append_version(args->out_hdr, args->argc, args->argv, "bcftools_norm");
if ( bcf_hdr_write(args->out, args->out_hdr)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out,args->out_hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out,args->out_hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
- bcf1_t *line;
- int prev_rid = -1, prev_pos = -1, prev_type = 0;
- while ( (line = next_atomized_line(args)) )
+ while (1)
{
- args->ntotal++;
- if ( args->rmdup )
- {
- int line_type = bcf_get_variant_types(line);
- if ( prev_rid>=0 && prev_rid==line->rid && prev_pos==line->pos )
- {
- if ( args->rmdup & BCF_SR_PAIR_ANY ) continue; // rmdup by position only
- if ( args->rmdup & BCF_SR_PAIR_SNPS && line_type&(VCF_SNP|VCF_MNP) && prev_type&(VCF_SNP|VCF_MNP) ) continue;
- if ( args->rmdup & BCF_SR_PAIR_INDELS && line_type&(VCF_INDEL) && prev_type&(VCF_INDEL) ) continue;
- if ( args->rmdup & BCF_SR_PAIR_EXACT && cmpals_match(&args->cmpals_in, line) ) continue;
- }
- else
- {
- prev_rid = line->rid;
- prev_pos = line->pos;
- prev_type = 0;
- if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_reset(&args->cmpals_in);
- }
- prev_type |= line_type;
- if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_add(&args->cmpals_in, line);
- }
-
- // still on the same chromosome?
- int i,j,ilast = rbuf_last(&args->rbuf);
- if ( ilast>=0 && line->rid != args->lines[ilast]->rid ) flush_buffer(args, args->out, args->rbuf.n); // new chromosome
-
- int split = 0;
- if ( args->mrows_op==MROWS_SPLIT )
+ // buffer lines until a different position is reached
+ int done = 0;
+ while (1)
{
- split = 1;
- if ( args->mrows_collapse!=COLLAPSE_BOTH && args->mrows_collapse!=COLLAPSE_ANY )
- {
- if ( !(bcf_get_variant_types(line) & args->mrows_collapse) ) split = 0;
- }
- if ( split && line->n_allele>2 )
- {
- args->nsplit++;
- split_multiallelic_to_biallelics(args, line);
- for (j=0; j<args->ntmp_lines; j++)
- normalize_line(args, args->tmp_lines[j]);
- }
- else
- split = 0;
+ done = split_and_normalize(args);
+ if ( done ) break; // no more lines available
+ int i = args->rbuf.f;
+ int j = rbuf_last(&args->rbuf);
+ if ( args->lines[i]->rid != args->lines[j]->rid ) break;
+ if ( args->lines[i]->pos != args->lines[j]->pos ) break;
}
- if ( !split )
- normalize_line(args, line);
+ if ( done ) break;
// find out how many sites to flush
- ilast = rbuf_last(&args->rbuf);
- j = 0;
+ int ifst = args->rbuf.f;
+ int ilast = rbuf_last(&args->rbuf);
+ int i, j = 0;
for (i=-1; rbuf_next(&args->rbuf,&i); )
{
+ if ( args->lines[ifst]->rid != args->lines[ilast]->rid )
+ {
+ // there are two chromosomes in the buffer, count how many are on the first chromosome
+ if ( args->lines[ifst]->rid != args->lines[i]->rid ) break;
+ j++;
+ continue;
+ }
+ // there is just one chromosome, flush only lines that are unlikely to change order on
+ // realigning (the buf_win constant)
if ( args->lines[ilast]->pos - args->lines[i]->pos < args->buf_win ) break;
j++;
}
}
if ( hts_close(args->out)!=0 ) error("[%s] Error: close failed .. %s\n", __func__,args->output_fname);
- fprintf(stderr,"Lines total/split/realigned/skipped:\t%d/%d/%d/%d\n", args->ntotal,args->nsplit,args->nchanged,args->nskipped);
+ fprintf(stderr,"Lines total/split/joined/realigned/removed/skipped:\t%d/%d/%d/%d/%d/%d\n",
+ args->ntotal,args->nsplit,args->njoined,args->nchanged,args->nskipped,args->nfilter);
if ( args->check_ref & CHECK_REF_FIX )
fprintf(stderr,"REF/ALT total/modified/added: \t%d/%d/%d\n", args->nref.tot,args->nref.swap,args->nref.set);
}
fprintf(stderr, " -c, --check-ref e|w|x|s Check REF alleles and exit (e), warn (w), exclude (x), or set (s) bad sites [e]\n");
fprintf(stderr, " -D, --remove-duplicates Remove duplicate lines of the same type.\n");
fprintf(stderr, " -d, --rm-dup TYPE Remove duplicate snps|indels|both|all|exact\n");
+ fprintf(stderr, " -e, --exclude EXPR Do not normalize records for which the expression is true (see man page for details)\n");
fprintf(stderr, " -f, --fasta-ref FILE Reference sequence\n");
fprintf(stderr, " --force Try to proceed even if malformed tags are encountered. Experimental, use at your own risk\n");
fprintf(stderr, " -g, --gff-annot FILE Follow HGVS 3'rule and right-align variants in transcripts on the forward strand\n");
+ fprintf(stderr, " -i, --include EXPR Normalize only records for which the expression is true (see man page for details)\n");
fprintf(stderr, " --keep-sum TAG,.. Keep vector sum constant when splitting multiallelics (see github issue #360)\n");
fprintf(stderr, " -m, --multiallelics -|+TYPE Split multiallelics (-) or join biallelics (+), type: snps|indels|both|any [both]\n");
fprintf(stderr, " --multi-overlaps 0|. Fill in the reference (0) or missing (.) allele when splitting multiallelics [0]\n");
fprintf(stderr, " -R, --regions-file FILE Restrict to regions listed in a file\n");
fprintf(stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
fprintf(stderr, " -s, --strict-filter When merging (-m+), merged site is PASS only if all sites being merged PASS\n");
+ fprintf(stderr, " -S, --sort METHOD Sort order: chr_pos,lex [chr_pos]\n");
fprintf(stderr, " -t, --targets REGION Similar to -r but streams rather than index-jumps\n");
fprintf(stderr, " -T, --targets-file FILE Similar to -R but streams rather than index-jumps\n");
fprintf(stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
- fprintf(stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
+ fprintf(stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
+ fprintf(stderr, " -v, --verbose INT Verbosity level (0-2) of GFF parsing [1]\n");
fprintf(stderr, " -w, --site-win INT Buffer for sorting lines which changed position during realignment [1000]\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Examples:\n");
fprintf(stderr, " # normalize and left-align indels\n");
int region_is_file = 0;
int targets_is_file = 0;
args->use_star_allele = 1;
+ args->gff_verbosity = 1;
int regions_overlap = 1;
int targets_overlap = 0;
+ args->cmp_func = cmp_bcf_pos;
static struct option loptions[] =
{
{"force",no_argument,NULL,7},
{"atomize",no_argument,NULL,'a'},
{"atom-overlaps",required_argument,NULL,11},
+ {"include",required_argument,NULL,'i'},
+ {"exclude",required_argument,NULL,'e'},
{"old-rec-tag",required_argument,NULL,12},
{"keep-sum",required_argument,NULL,10},
{"fasta-ref",required_argument,NULL,'f'},
+ {"sort",required_argument,NULL,'S'},
{"gff-annot",required_argument,NULL,'g'},
{"right-align",no_argument,NULL,15}, // undocumented, only for debugging
{"do-not-normalize",no_argument,NULL,'N'},
{"check-ref",required_argument,NULL,'c'},
{"strict-filter",no_argument,NULL,'s'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,14},
+ {"write-index",optional_argument,NULL,'W'},
+ {"verbose",required_argument,NULL,'v'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hr:R:f:w:Dd:o:O:c:m:t:T:sNag:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hr:R:f:w:Dd:o:O:c:m:t:T:sNag:W::v:S:i:e:",loptions,NULL)) >= 0) {
switch (c) {
case 10:
// possibly generalize this also to INFO/AD and other tags
args->keep_sum_ad = 1; // this will be set to the header id or -1 in init_data
break;
case 'g': args->gff_fname = optarg; break;
+ case 'v':
+ args->gff_verbosity = atoi(optarg);
+ if ( args->gff_verbosity<0 || args->gff_verbosity>2 ) error("Error: expected integer 0-2 with -v, --verbose\n");
+ break;
case 'a': args->atomize = SPLIT; break;
+ case 'e':
+ if ( args->filter_str ) error("Error: only one -i or -e expression can be given, and they cannot be combined\n");
+ args->filter_str = optarg; args->filter_logic |= FLT_EXCLUDE; break;
+ case 'i':
+ if ( args->filter_str ) error("Error: only one -i or -e expression can be given, and they cannot be combined\n");
+ args->filter_str = optarg; args->filter_logic |= FLT_INCLUDE; break;
+ case 'S':
+ if ( !strcasecmp(optarg,"pos") ) args->cmp_func = cmp_bcf_pos;
+ else if ( !strcasecmp(optarg,"lex") ) args->cmp_func = cmp_bcf_pos_ref_alt;
+ else error("Error: the sort order --sort %s is not recognised\n",optarg);
+ break;
case 11 :
if ( optarg[0]=='*' ) args->use_star_allele = 1;
else if ( optarg[0]=='.' ) args->use_star_allele = 0;
else if ( optarg[0]=='.' ) args->ma_use_ref_allele = 0;
else error("Invalid argument to --multi-overlaps\n");
break;
- case 14 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 15 : args->right_align = 1; break;
case 'N': args->do_indels = 0; break;
case 'd':
if ( bcf_sr_set_threads(args->files, args->n_threads)<0 ) error("Failed to create threads\n");
if ( !bcf_sr_add_reader(args->files, fname) ) error("Failed to read from %s: %s\n", !strcmp("-",fname)?"standard input":fname,bcf_sr_strerror(args->files->errnum));
- if ( args->mrows_op&MROWS_SPLIT && args->rmdup ) error("Cannot combine -D and -m-\n");
init_data(args);
normalize_vcf(args);
destroy_data(args);
/* vcfnorm.c -- Left-align and normalize indels.
- Copyright (C) 2013-2023 Genome Research Ltd.
+ Copyright (C) 2013-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include "abuf.h"
#include "gff.h"
#include "regidx.h"
+#include "filter.h"
#define CHECK_REF_EXIT 1
#define CHECK_REF_WARN 2
#define MROWS_SPLIT 1
#define MROWS_MERGE 2
+// Logic of the filters: include or exclude sites which match the filters?
+#define FLT_INCLUDE 1
+#define FLT_EXCLUDE 2
+
// for -m+, mapping from allele indexes of a single input record
// to allele indexes of output record
typedef struct
{
int n; // number of alleles
char *ref, *alt;
- void *hash;
+ void *hash; // str2int hash
}
cmpals1_t;
{
char *tseq, *seq;
int mseq;
- bcf1_t **lines, **tmp_lines, **alines, **blines, *mrow_out;
- int ntmp_lines, mtmp_lines, nalines, malines, nblines, mblines;
+ bcf1_t **lines, **tmp_lines, **mrows, *mrow_out;
+ int ntmp_lines, mtmp_lines, nmrows, mmrows, mrows_first;
map_t *maps; // mrow map for each buffered record
char **als;
int mmaps, nals, mals;
int32_t *int32_arr;
int ntmp_arr1, ntmp_arr2, nint32_arr;
kstring_t *tmp_str;
- kstring_t *tmp_als, *tmp_del, tmp_kstr;
- int ntmp_als, ntmp_del;
+ kstring_t *tmp_als, *tmp_sym, tmp_kstr;
+ int ntmp_als, ntmp_sym;
rbuf_t rbuf;
int buf_win; // maximum distance between two records to consider
int aln_win; // the realignment window size (maximum repeat size)
struct { int tot, set, swap; } nref;
char **argv, *output_fname, *ref_fname, *vcf_fname, *region, *targets;
int argc, rmdup, output_type, n_threads, check_ref, strict_filter, do_indels, clevel;
- int nchanged, nskipped, nsplit, ntotal, mrows_op, mrows_collapse, parsimonious;
+ int nchanged, nskipped, nsplit, njoined, ntotal, nfilter, mrows_op, mrows_collapse, parsimonious;
int record_cmd_line, force, force_warned, keep_sum_ad;
abuf_t *abuf;
abuf_opt_t atomize;
char *old_rec_tag;
htsFile *out;
char *index_fn;
- int write_index;
+ int write_index, gff_verbosity;
int right_align;
char *gff_fname;
gff_t *gff;
regidx_t *idx_tscript;
regitr_t *itr_tscript;
+ int (*cmp_func)(const void *aptr, const void *bptr);
+ char *filter_str;
+ int filter_logic; // include or exclude sites which match the filters? One of FLT_INCLUDE/FLT_EXCLUDE
+ int filter_pass;
+ filter_t *filter;
}
args_t;
if ( bcf_get_variant_types(line)==VCF_BND ) return ERR_SYMBOLIC; // breakend, not an error
// make a copy of each allele for trimming
- hts_expand0(kstring_t,line->n_allele,args->ntmp_als,args->tmp_als);
- hts_expand0(kstring_t,line->n_allele,args->ntmp_del,args->tmp_del);
+ hts_expand0(kstring_t,line->n_allele,args->ntmp_als,args->tmp_als); // the actual sequence to realign
+ hts_expand0(kstring_t,line->n_allele,args->ntmp_sym,args->tmp_sym); // the original symbolic allele strings to output
kstring_t *als = args->tmp_als;
- kstring_t *del = args->tmp_del;
+ kstring_t *sym = args->tmp_sym;
+ int symbolic_alts = 1;
for (i=0; i<line->n_allele; i++)
{
- del[i].l = 0;
+ sym[i].l = 0;
if ( line->d.allele[i][0]=='<' )
{
- // symbolic allele, only <DEL.*> will be realigned
- if ( strncmp("<DEL",line->d.allele[i],4) ) return ERR_SYMBOLIC;
- if ( nref < line->rlen )
+ // symbolic allele, only <DEL.*> and <DUP.*> will be realigned
+ // TODO: there should be check for symbolic allele length. If too big, perhaps should not attempt realignment
+ int32_t sv_len = 0;
+ if ( !strncmp("<DEL",line->d.allele[i],4) ) sv_len = -line->rlen;
+ else if ( !strncmp("<DUP",line->d.allele[i],4) )
+ {
+ if ( bcf_get_info_int32(args->hdr,line,"SVLEN",&args->int32_arr,&args->nint32_arr)==1 ) sv_len = args->int32_arr[0];
+ }
+ if ( !sv_len ) return ERR_SYMBOLIC;
+
+ als[i].l = 0;
+ if ( sv_len<0 )
+ {
+ // del, expand REF and replace ALT, for example, replace "REF=C ALT=<DEL>" with "REF=CAT ALT=C"
+ if ( nref < line->rlen )
+ {
+ free(ref);
+ reflen = line->rlen;
+ ref = faidx_fetch_seq(args->fai, (char*)args->hdr->id[BCF_DT_CTG][line->rid].key, line->pos, line->pos+reflen-1, &nref);
+ if ( !ref ) error("faidx_fetch_seq failed at %s:%"PRId64"\n", args->hdr->id[BCF_DT_CTG][line->rid].key, (int64_t) line->pos+1);
+ seq_to_upper(ref,0);
+ replace_iupac_codes(ref,nref); // any non-ACGT character in fasta ref is replaced with N
+ als[0].l = 0;
+ kputs(ref, &als[0]);
+ }
+ kputsn(als[0].s,1,&als[i]);
+ }
+ else // sv_len>0
{
+ // dup, replace "REF=C ALT=<DUP>" with "REF=C ALT=CAT"
free(ref);
- reflen = line->rlen;
- ref = faidx_fetch_seq(args->fai, (char*)args->hdr->id[BCF_DT_CTG][line->rid].key, line->pos, line->pos+reflen-1, &nref);
+ ref = faidx_fetch_seq(args->fai, (char*)args->hdr->id[BCF_DT_CTG][line->rid].key, line->pos, line->pos+sv_len, &nref);
if ( !ref ) error("faidx_fetch_seq failed at %s:%"PRId64"\n", args->hdr->id[BCF_DT_CTG][line->rid].key, (int64_t) line->pos+1);
seq_to_upper(ref,0);
replace_iupac_codes(ref,nref); // any non-ACGT character in fasta ref is replaced with N
- als[0].l = 0;
- kputs(ref, &als[0]);
- als[i].l = 0;
- kputsn(ref,1,&als[i]);
- kputs(line->d.allele[i],&del[i]);
- continue;
+ kputs(ref,&als[i]);
}
+ kputs(line->d.allele[i],&sym[i]); // preserve the symbolic allele string
+ continue;
}
+ if ( i>0 ) symbolic_alts = 0;
if ( line->d.allele[i][0]=='*' ) return ERR_SPANNING_DELETION; // spanning deletion
if ( has_non_acgtn(line->d.allele[i],line->shared.l) )
{
else
new_pos = realign_right(args, line);
- // Have the alleles changed?
- als[0].s[ als[0].l ] = 0; // in order for strcmp to work
+ // Have the alleles changed? Consider <DEL> could have expanded the REF allele. In that
+ // case it must be trimmed, however the new REF length must reflect the entire length.
+ als[0].s[ als[0].l ] = 0; // for strcmp to work
+ int new_reflen = strlen(als[0].s);
+ if ( symbolic_alts )
+ {
+ als[0].l = 1;
+ als[0].s[ als[0].l ] = 0;
+ }
if ( new_pos==line->pos && !strcasecmp(line->d.allele[0],als[0].s) ) return ERR_OK;
set_old_rec_tag(args, line, line, 0);
for (i=0; i<line->n_allele; i++)
{
if (i>0) kputc(',',&args->tmp_kstr);
- if ( del[i].l ) kputs(del[i].s,&args->tmp_kstr);
+ if ( sym[i].l ) kputs(sym[i].s,&args->tmp_kstr);
else kputsn(als[i].s,als[i].l,&args->tmp_kstr);
}
args->tmp_kstr.s[ args->tmp_kstr.l ] = 0;
args->nchanged++;
// Update INFO/END if necessary
- int new_reflen = strlen(line->d.allele[0]);
if ( (new_pos!=line->pos || reflen!=new_reflen) && bcf_get_info_int32(args->hdr, line, "END", &args->int32_arr, &args->nint32_arr)==1 )
{
// bcf_update_alleles_str() messed up rlen because line->pos changed. This will be fixed by bcf_update_info_int32()
bcf_update_info_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele-1,ret); \
} \
bcf_update_info_##type(args->out_hdr,dst,tag,vals+ialt,1); \
bcf_update_info_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele,ret); \
} \
if ( args->keep_sum_ad >= 0 && args->keep_sum_ad==info->key ) \
bcf_update_info_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in INFO/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele*(src->n_allele+1)/2,ret); \
} \
if ( ialt!=0 ) \
}
#undef BRANCH_NUMERIC
}
-// Find n-th field in a comma-separated list and move it to dst.
-// The memory areas may overlap.
-#define STR_MOVE_NTH(dst,src,end,nth,len) \
-{ \
- char *ss = src, *se = src; \
- int j = 0; \
- while ( *se && se<(end) ) \
- { \
- if ( *se==',' ) \
- { \
- if ( j==nth ) break; \
- j++; \
- ss = se+1; \
- } \
- se++; \
- } \
- if ( j==nth ) \
- { \
- int n = se - ss; \
- memmove((dst),ss,n); \
- src = se; \
- len += n; \
- } \
- else len = -1; \
+// Find nth field in a comma-separated list in src and move it to dst.
+// The dst and src memory areas may overlap, end points just after the last valid src
+// character.
+// On success returns pointer to the end of the parsed field and increments ndst by the
+// number of memmoved characters. Returns NULL if the field was not found.
+static inline char *string_move_nth(char *dst, char *src, char *end, int nth, size_t *ndst)
+{
+ if ( src>=end ) return NULL;
+ char *ss = src, *se = src;
+ int j = 0;
+ while ( *se && se<(end) )
+ {
+ if ( *se==',' )
+ {
+ if ( j==nth ) break;
+ j++;
+ ss = se+1;
+ }
+ se++;
+ }
+ if ( j!=nth ) return NULL;
+ if ( ss>=end ) return NULL;
+ if ( !*ss ) return NULL;
+
+ int n = se - ss;
+ memmove((dst),ss,n);
+ *ndst += n;
+ return se;
}
+
static void split_info_string(args_t *args, bcf1_t *src, bcf_info_t *info, int ialt, bcf1_t *dst)
{
const char *tag = bcf_hdr_int2id(args->hdr,BCF_DT_ID,info->key);
int len = bcf_hdr_id2length(args->hdr,BCF_HL_INFO,info->key);
if ( len==BCF_VL_A )
{
+ char *end = str.s + str.l;
char *tmp = str.s;
- int len = 0;
- STR_MOVE_NTH(str.s,tmp,str.s+str.l,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len] = 0;
+ str.l = 0;
+ tmp = string_move_nth(str.s,tmp,end,ialt,&str.l);
+ if ( !tmp ) str.l = 1, str.s[0] = '.';
+ kputc_(0,&str);
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
}
else if ( len==BCF_VL_R )
{
+ char *end = str.s + str.l;
char *tmp = str.s;
- int len = 0;
- STR_MOVE_NTH(str.s,tmp,str.s+str.l,0,len);
- str.s[len]=','; tmp++; len++;
- STR_MOVE_NTH(&str.s[len],tmp,str.s+str.l,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len] = 0;
+ str.l = 0;
+ tmp = string_move_nth(str.s,tmp,end,0,&str.l);
+ if ( tmp )
+ {
+ kputc_(',',&str);
+ tmp = string_move_nth(str.s+str.l,tmp+1,end,ialt,&str.l); // ialt is 0-based index to ALT
+ }
+ if ( !tmp ) str.l = 1, str.s[0] = '.';
+ kputc_(0,&str);
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
}
else if ( len==BCF_VL_G )
{
int i0a = bcf_alleles2gt(0,ialt+1), iaa = bcf_alleles2gt(ialt+1,ialt+1);
+ char *end = str.s + str.l;
char *tmp = str.s;
- int len = 0;
- STR_MOVE_NTH(str.s,tmp,str.s+str.l,0,len);
- str.s[len]=','; tmp++; len++;
- STR_MOVE_NTH(&str.s[len],tmp,str.s+str.l,i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len]=','; tmp++; len++;
- STR_MOVE_NTH(&str.s[len],tmp,str.s+str.l,iaa-i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
- str.s[len] = 0;
+ str.l = 0;
+ tmp = string_move_nth(str.s,tmp,end,0,&str.l);
+ if ( tmp )
+ {
+ kputc_(',',&str);
+ tmp = string_move_nth(str.s+str.l,tmp+1,end,i0a-1,&str.l);
+ }
+ if ( tmp )
+ {
+ kputc_(',',&str);
+ tmp = string_move_nth(str.s+str.l,tmp+1,end,iaa-i0a-1,&str.l);
+ }
+ if ( !tmp ) str.l = 1, str.s[0] = '.';
+ kputc_(0,&str);
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
}
else
bcf_update_info_string(args->out_hdr,dst,tag,str.s);
+ if ( args->ntmp_arr1 < str.m )
+ {
+ args->ntmp_arr1 = str.m;
+ args->tmp_arr1 = (uint8_t*)str.s;
+ }
}
static void split_info_flag(args_t *args, bcf1_t *src, bcf_info_t *info, int ialt, bcf1_t *dst)
{
{
if ( gt[j]==bcf_int32_vector_end ) break;
if ( bcf_gt_is_missing(gt[j]) ) continue; // missing allele: leave as is
- if ( (ialt==0 || args->ma_use_ref_allele) && bcf_gt_allele(gt[j])==0 ) continue; // ref && `--multi-overlaps 0`: leave as is
+ if ( bcf_gt_allele(gt[j])==0 ) continue; // ref && `--multi-overlaps 0`: leave as is
if ( bcf_gt_allele(gt[j])==ialt+1 )
gt[j] = bcf_gt_unphased(1) | bcf_gt_is_phased(gt[j]); // set to first ALT
else if ( args->ma_use_ref_allele )
bcf_update_format_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,(src->n_allele-1)*nsmpl,nvals); \
} \
nvals /= nsmpl; \
bcf_update_format_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d\n", \
+ error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d, found %d. Use --force to proceed anyway.\n", \
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele*nsmpl,nvals); \
} \
nvals /= nsmpl; \
bcf_update_format_##type(args->out_hdr,dst,tag,NULL,0); \
return; \
} \
- error("Error at %s:%"PRId64", the tag %s has wrong number of fields\n", bcf_seqname(args->hdr,src),(int64_t) src->pos+1,bcf_hdr_int2id(args->hdr,BCF_DT_ID,fmt->id)); \
+ error("Error at %s:%"PRId64", the tag %s has wrong number of fields. Use --force to proceed anyway.\n", \
+ bcf_seqname(args->hdr,src),(int64_t) src->pos+1,bcf_hdr_int2id(args->hdr,BCF_DT_ID,fmt->id)); \
} \
nvals /= nsmpl; \
int all_haploid = nvals==src->n_allele ? 1 : 0; \
isrc += src_blen;
}
}
+// ialt is 0-based index to ALT
static void split_format_string(args_t *args, bcf1_t *src, bcf_fmt_t *fmt, int ialt, bcf1_t *dst)
{
const char *tag = bcf_hdr_int2id(args->hdr,BCF_DT_ID,fmt->id);
if ( !ret ) return; // all values can be empty, leave out the tag, no need to panic
assert( ret>0 );
+ int nsmpl = bcf_hdr_nsamples(args->hdr);
+ int blen = ret/nsmpl; // per-sample field length
+ assert( blen>0 );
+
kstring_t str;
str.m = args->ntmp_arr1;
- str.l = ret;
str.s = (char*) args->tmp_arr1;
+ str.l = ret;
- int nsmpl = bcf_hdr_nsamples(args->hdr);
- int len = bcf_hdr_id2length(args->hdr,BCF_HL_FMT,fmt->id);
- if ( len==BCF_VL_A )
+ int tag_len = bcf_hdr_id2length(args->hdr,BCF_HL_FMT,fmt->id);
+ if ( tag_len==BCF_VL_A )
{
- int i, blen = ret/nsmpl, maxlen = 0;
+ int i, maxlen = 0;
char *ptr = str.s;
for (i=0; i<nsmpl; i++)
{
char *tmp = ptr;
- int len = 0;
- STR_MOVE_NTH(tmp,tmp,ptr+blen,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ char *end = ptr + blen;
+ size_t len = 0;
+ tmp = string_move_nth(ptr,tmp,end,ialt,&len);
+ if ( !tmp ) ptr[0] = '.', len = 1;
if ( maxlen < len ) maxlen = len;
+ while (len<blen) ptr[len++] = 0;
ptr += blen;
}
if ( maxlen<blen ) squeeze_format_char(str.s,blen,maxlen,nsmpl);
bcf_update_format_char(args->out_hdr,dst,tag,str.s,nsmpl*maxlen);
}
- else if ( len==BCF_VL_R )
+ else if ( tag_len==BCF_VL_R )
{
- int i, blen = ret/nsmpl, maxlen = 0;
+ int i, maxlen = 0;
char *ptr = str.s;
for (i=0; i<nsmpl; i++)
{
char *tmp = ptr;
- int len = 0;
- STR_MOVE_NTH(ptr,tmp,ptr+blen,0,len);
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ char *end = ptr + blen;
+ size_t len = 0;
+ tmp = string_move_nth(ptr,tmp,end,0,&len);
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,ialt,&len);
+ }
+ if ( !tmp ) ptr[0] = '.', len = 1;
if ( maxlen < len ) maxlen = len;
+ while (len<blen) ptr[len++] = 0;
ptr += blen;
}
if ( maxlen<blen ) squeeze_format_char(str.s,blen,maxlen,nsmpl);
bcf_update_format_char(args->out_hdr,dst,tag,str.s,nsmpl*maxlen);
}
- else if ( len==BCF_VL_G )
+ else if ( tag_len==BCF_VL_G )
{
- int i, blen = ret/nsmpl, maxlen = 0, i0a = bcf_alleles2gt(0,ialt+1), iaa = bcf_alleles2gt(ialt+1,ialt+1);
+ int i, maxlen = 0, i0a = bcf_alleles2gt(0,ialt+1), iaa = bcf_alleles2gt(ialt+1,ialt+1);
char *ptr = str.s;
for (i=0; i<nsmpl; i++)
{
bcf_update_format_char(args->out_hdr,dst,tag,NULL,0);
return;
}
- error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d or %d, found %d\n",
+ error("Error: wrong number of fields in FMT/%s at %s:%"PRId64", expected %d or %d, found %d. Use --force to proceed anyway.\n",
tag,bcf_seqname(args->hdr,src),(int64_t) src->pos+1,src->n_allele*(src->n_allele+1)/2,src->n_allele,nfields);
}
- int len = 0;
+ char *tmp = ptr;
+ char *end = ptr + blen;
+ size_t len = 0;
+ tmp = string_move_nth(ptr,tmp,end,0,&len);
if ( nfields==src->n_allele ) // haploid
{
- char *tmp = ptr;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,0,len);
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,ialt,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,ialt,&len);
+ }
}
else // diploid
{
- char *tmp = ptr;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,0,len);
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
- ptr[len]=','; tmp++; len++;
- STR_MOVE_NTH(&ptr[len],tmp,ptr+blen,iaa-i0a-1,len);
- if ( len<0 ) return; // wrong number of fields: skip
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,i0a-1,&len);
+ }
+ if ( tmp )
+ {
+ ptr[len++] = ',';
+ tmp = string_move_nth(ptr+len,tmp+1,end,iaa-i0a-1,&len);
+ }
}
+ if ( !tmp ) ptr[0] = '.', len = 1;
if ( maxlen < len ) maxlen = len;
+ while (len<blen) ptr[len++] = 0;
ptr += blen;
}
if ( maxlen<blen ) squeeze_format_char(str.s,blen,maxlen,nsmpl);
bcf_update_info_string(args->out_hdr,dst,tag,args->tmp_arr1);
}
}
+static int gt_array_grow_ploidy(args_t *args, uint8_t **tmp_arr, int *ntmp_arr, int ngt_ori, int ngt_new, int nsmpl)
+{
+ *ntmp_arr = 4*ngt_new*nsmpl;
+ int32_t *ptr = (int32_t*)realloc(*tmp_arr,*ntmp_arr);
+ if ( !ptr ) error("Error: failed to allocate %d bytes\n",*ntmp_arr);
+ *tmp_arr = (uint8_t*) ptr;
+
+ int i,j;
+ for (i=nsmpl-1; i>=0; i--)
+ {
+ int32_t *src = ptr + i*ngt_ori;
+ int32_t *dst = ptr + i*ngt_new;
+ for (j=ngt_new; j>ngt_ori; j--) dst[j-1] = bcf_int32_vector_end;
+ for (j=ngt_ori; j>0; j--) dst[j-1] = src[j-1];
+ }
+ return ngt_new;
+}
static void merge_format_genotype(args_t *args, bcf1_t **lines, int nlines, bcf_fmt_t *fmt, bcf1_t *dst)
{
// reusing int8_t arrays as int32_t arrays
int ngts2 = bcf_get_genotypes(args->hdr,lines[i],&args->tmp_arr2,&ntmp2);
args->ntmp_arr2 = ntmp2 * 4;
ngts2 /= nsmpl;
- if ( ngts!=ngts2 ) error("Error at %s:%"PRId64": cannot combine diploid with haploid genotype\n", bcf_seqname(args->hdr,lines[i]),(int64_t) lines[i]->pos+1);
+ int ploidy_changed = ngts - ngts2;
+ if ( ngts < ngts2 ) ngts = gt_array_grow_ploidy(args,&args->tmp_arr1,&args->ntmp_arr1,ngts,ngts2,nsmpl);
+ if ( ngts > ngts2 ) ngts2 = gt_array_grow_ploidy(args,&args->tmp_arr2,&args->ntmp_arr2,ngts2,ngts,nsmpl);
int32_t *gt = (int32_t*) args->tmp_arr1; // the first, destination line
int32_t *gt2 = (int32_t*) args->tmp_arr2; // one of the subsequent lines, i.e. the source line
// never overwrite with ref allele
for (k2=0; k2<ngts2; k2++)
{
- if ( gt2[k2]==bcf_int32_vector_end ) break;
- if ( bcf_gt_is_missing(gt2[k2]) ) continue;
+ if ( gt2[k2]==bcf_int32_vector_end )
+ {
+ if ( ploidy_changed && bcf_gt_is_missing(gt[k2]) ) gt[k2] = bcf_int32_vector_end;
+ break;
+ }
+ if ( bcf_gt_is_missing(gt2[k2]) ) continue; // don't overwrite with missing
+
+ // don't overwrite with ref, unless the destination is missing, e.g. "./. + 0/1"
int ial2 = bcf_gt_allele(gt2[k2]);
- if ( ial2==0 ) continue; // never overwrite with ref
+ if ( ial2==0 && !bcf_gt_is_missing(gt[k2]) && gt[k2]!=bcf_int32_vector_end ) continue;
if ( ial2>=args->maps[i].nals ) error("Error at %s:%"PRId64": incorrect allele index %d\n",bcf_seqname(args->hdr,lines[i]),(int64_t) lines[i]->pos+1,ial2);
// The destination allele
int ial = args->maps[i].map[ial2];
if ( gt[k2]==bcf_int32_vector_end || bcf_gt_is_missing(gt[k2]) || !bcf_gt_allele(gt[k2]) )
- gt[k2] = bcf_gt_is_phased(gt[k2]) ? bcf_gt_phased(ial) : bcf_gt_unphased(ial);
+ gt[k2] = (gt[k2]!=bcf_int32_vector_end && bcf_gt_is_phased(gt[k2])) ? bcf_gt_phased(ial) : bcf_gt_unphased(ial);
else
{
// conflict, the first line has non-zero allele, use the old way, possibly disrupt the phasing
else if ( type==BCF_HT_INT || type==BCF_HT_REAL ) merge_format_numeric(args, lines, nlines, fmt, dst);
else merge_format_string(args, lines, nlines, fmt, dst);
}
+ args->njoined++;
}
#define SWAP(type_t, a, b) { type_t t = a; a = b; b = t; }
-static void mrows_schedule(args_t *args, bcf1_t **line)
+static void mrows_push(args_t *args, bcf1_t **line)
{
int i,m;
- if ( args->mrows_collapse==COLLAPSE_ANY // merge all record types together
- || bcf_get_variant_types(*line)&VCF_SNP // SNP, put into alines
- || bcf_get_variant_types(*line)==VCF_REF ) // ref
- {
- args->nalines++;
- m = args->malines;
- hts_expand(bcf1_t*,args->nalines,args->malines,args->alines);
- for (i=m; i<args->malines; i++) args->alines[i] = bcf_init1();
- SWAP(bcf1_t*, args->alines[args->nalines-1], *line);
- }
- else
- {
- args->nblines++;
- m = args->mblines;
- hts_expand(bcf1_t*,args->nblines,args->mblines,args->blines);
- for (i=m; i<args->mblines; i++) args->blines[i] = bcf_init1();
- SWAP(bcf1_t*, args->blines[args->nblines-1], *line);
+ if ( !args->nmrows ) args->mrows_first = 0;
+ args->nmrows++;
+ m = args->mmrows;
+ hts_expand(bcf1_t*,args->nmrows,args->mmrows,args->mrows);
+ for (i=m; i<args->mmrows; i++) args->mrows[i] = bcf_init1();
+ SWAP(bcf1_t*, args->mrows[args->nmrows-1], *line);
+
+ if ( args->mrows_collapse==COLLAPSE_ANY ) return;
+
+ // move the line up the sorted list so that the same variant types end up together
+ int cur_type = bcf_get_variant_types(args->mrows[args->nmrows-1]);
+ i = args->mrows_first + args->nmrows - 1;
+ while (i>0)
+ {
+ int prev_type = bcf_get_variant_types(args->mrows[i-1]);
+ if ( prev_type <= cur_type ) break;
+ bcf1_t *tmp = args->mrows[i-1];
+ args->mrows[i-1] = args->mrows[i];
+ args->mrows[i] = tmp;
+ i--;
}
}
-static int mrows_ready_to_flush(args_t *args, bcf1_t *line)
+static int mrows_can_flush(args_t *args, bcf1_t *line)
{
- if ( args->nalines && (args->alines[0]->rid!=line->rid || args->alines[0]->pos!=line->pos) ) return 1;
- if ( args->nblines && (args->blines[0]->rid!=line->rid || args->blines[0]->pos!=line->pos) ) return 1;
+ if ( !args->nmrows ) return 0;
+ int ibeg = args->mrows_first;
+ if ( args->mrows[ibeg]->rid != line->rid ) return 1;
+ if ( args->mrows[ibeg]->pos != line->pos ) return 1;
return 0;
}
static bcf1_t *mrows_flush(args_t *args)
{
- if ( args->nblines && args->nalines==1 && bcf_get_variant_types(args->alines[0])==VCF_REF )
+ if ( !args->nmrows ) return NULL;
+
+ int ibeg = args->mrows_first;
+
+ //fprintf(bcftools_stderr,"flush: ibeg=%d n=%d\n",ibeg,args->nmrows);
+ //int i;
+ //for (i=ibeg; i<ibeg+args->nmrows; i++)
+ // fprintf(bcftools_stderr,"\ti=%d type=%d %s %s\n",i,bcf_get_variant_types(args->mrows[i]),args->mrows[i]->d.allele[0],args->mrows[i]->d.allele[1]);
+
+ if ( args->nmrows==1 )
{
- // By default, REF lines are merged with SNPs if SNPs and indels are to be kept separately.
- // However, if there are indels only and a single REF line, merge it with indels.
- args->nblines++;
- int i,m = args->mblines;
- hts_expand(bcf1_t*,args->nblines,args->mblines,args->blines);
- for (i=m; i<args->mblines; i++) args->blines[i] = bcf_init1();
- SWAP(bcf1_t*, args->blines[args->nblines-1], args->alines[0]);
- args->nalines--;
+ args->nmrows = 0;
+ return args->mrows[ibeg];
}
- if ( args->nalines )
+
+ if ( args->mrows_collapse==COLLAPSE_ANY )
{
- if ( args->nalines==1 )
- {
- args->nalines = 0;
- return args->alines[0];
- }
+ // merge everything with anything
bcf_clear(args->mrow_out);
- merge_biallelics_to_multiallelic(args, args->mrow_out, args->alines, args->nalines);
- args->nalines = 0;
+ merge_biallelics_to_multiallelic(args, args->mrow_out, &args->mrows[ibeg], args->nmrows - ibeg);
+ args->nmrows = 0;
return args->mrow_out;
}
- else if ( args->nblines )
+
+ int j;
+ int types[] = { VCF_SNP, VCF_MNP, VCF_INDEL, VCF_OTHER, -1 }; // merge everything within the same category
+ if ( args->mrows_collapse==COLLAPSE_SNPS ) types[1] = -1; // merge SNPs only
+ else if ( args->mrows_collapse==COLLAPSE_INDELS ) types[0] = VCF_INDEL, types[1] = -1; // merge indels only
+ for (j=0; types[j]!=-1; j++)
{
- if ( args->nblines==1 )
+ int i, type = types[j]; // to keep the compiler happy
+ for (i=ibeg; i<ibeg+args->nmrows; i++)
{
- args->nblines = 0;
- return args->blines[0];
+ type = bcf_get_variant_types(args->mrows[i]);
+ if ( type!=types[j] && type!=VCF_REF ) break;
+ }
+ if ( i==ibeg+1 && type!=VCF_REF )
+ {
+ // just one line of this type, no merging, but multiple lines of different type follow
+ args->nmrows--;
+ args->mrows_first++;
+ return args->mrows[ibeg];
+ }
+ if ( i>ibeg )
+ {
+ // more than one line, merging is needed
+ int nflush = i - ibeg;
+ bcf_clear(args->mrow_out);
+ merge_biallelics_to_multiallelic(args, args->mrow_out, &args->mrows[ibeg], nflush);
+ args->nmrows -= nflush;
+ args->mrows_first += nflush;
+ return args->mrow_out;
}
- bcf_clear(args->mrow_out);
- merge_biallelics_to_multiallelic(args, args->mrow_out, args->blines, args->nblines);
- args->nblines = 0;
- return args->mrow_out;
}
- return NULL;
+ args->nmrows--;
+ args->mrows_first++;
+ return args->mrows[ibeg];
}
-static void cmpals_add(cmpals_t *ca, bcf1_t *rec)
+static char *strdup_alt_svlen(args_t *args, bcf1_t *rec, int ial)
+{
+ if ( rec->d.allele[ial][0]!='<' ) return strdup(rec->d.allele[ial]);
+
+ int ntmp = args->ntmp_arr1 / sizeof(int32_t);
+ int n = bcf_get_info_int32(args->hdr, rec, "SVLEN", &args->tmp_arr1, &ntmp);
+ args->ntmp_arr1 = ntmp * sizeof(int32_t);
+ int32_t *svlen = (int32_t *) args->tmp_arr1;
+ if ( n<=0 ) return strdup(rec->d.allele[ial]);
+
+ if ( n+1 != rec->n_allele )
+ {
+ // there should be as many SVLEN numbers as there are ALT alleles
+ static int warned = 0;
+ if ( !warned )
+ {
+ fprintf(bcftools_stderr,"TODO: different number of ALT alleles and SVLEN fields %s:%"PRIhts_pos"\n",bcf_seqname(args->hdr,rec),rec->pos+1);
+ warned = 1;
+ }
+ }
+
+ kstring_t str = {0,0,0};
+ ksprintf(&str,"%s.%d",rec->d.allele[ial],svlen[ial-1]);
+ return str.s;
+}
+static void cmpals_add(args_t *args, cmpals_t *ca, bcf1_t *rec)
{
ca->ncmpals++;
hts_expand0(cmpals1_t, ca->ncmpals, ca->mcmpals, ca->cmpals);
free(cmpals->ref);
cmpals->ref = strdup(rec->d.allele[0]);
cmpals->n = rec->n_allele;
+
if ( rec->n_allele==2 )
{
free(cmpals->alt);
- cmpals->alt = strdup(rec->d.allele[1]);
+ cmpals->alt = strdup_alt_svlen(args,rec,1);
}
else
{
khash_str2int_inc(cmpals->hash, strdup(rec->d.allele[i]));
}
}
-static int cmpals_match(cmpals_t *ca, bcf1_t *rec)
+static int cmpals_match(args_t *args, cmpals_t *ca, bcf1_t *rec)
{
int i, j;
+ char *alt_svlen = rec->n_allele==2 ? strdup_alt_svlen(args,rec,1) : NULL;
for (i=0; i<ca->ncmpals; i++)
{
cmpals1_t *cmpals = ca->cmpals + i;
// the most frequent case
if ( rec->n_allele==2 )
{
- if ( strcasecmp(rec->d.allele[1], cmpals->alt) ) continue;
+ if ( strcasecmp(alt_svlen, cmpals->alt) ) continue;
+ free(alt_svlen);
return 1;
}
if ( j<rec->n_allele ) continue;
return 1;
}
+ free(alt_svlen);
return 0;
}
static void cmpals_reset(cmpals_t *ca) { ca->ncmpals = 0; }
k = rbuf_shift(&args->rbuf);
if ( args->mrows_op==MROWS_MERGE )
{
- if ( mrows_ready_to_flush(args, args->lines[k]) )
+ if ( mrows_can_flush(args, args->lines[k]) )
{
while ( (line=mrows_flush(args)) )
if ( bcf_write1(file, args->out_hdr, line)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
}
- int merge = 1;
- if ( args->mrows_collapse!=COLLAPSE_BOTH && args->mrows_collapse!=COLLAPSE_ANY )
- {
- if ( !(bcf_get_variant_types(args->lines[k]) & args->mrows_collapse) ) merge = 0;
- }
- if ( merge )
- {
- mrows_schedule(args, &args->lines[k]);
- continue;
- }
+ mrows_push(args, &args->lines[k]);
+ continue;
}
else if ( args->rmdup )
{
if ( args->rmdup & BCF_SR_PAIR_ANY ) continue; // rmdup by position only
if ( args->rmdup & BCF_SR_PAIR_SNPS && line_type&(VCF_SNP|VCF_MNP) && prev_type&(VCF_SNP|VCF_MNP) ) continue;
if ( args->rmdup & BCF_SR_PAIR_INDELS && line_type&(VCF_INDEL) && prev_type&(VCF_INDEL) ) continue;
- if ( args->rmdup & BCF_SR_PAIR_EXACT && cmpals_match(&args->cmpals_out, args->lines[k]) ) continue;
+ if ( args->rmdup & BCF_SR_PAIR_EXACT && cmpals_match(args, &args->cmpals_out, args->lines[k]) ) continue;
}
else
{
if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_reset(&args->cmpals_out);
}
prev_type |= line_type;
- if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_add(&args->cmpals_out, args->lines[k]);
+ if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_add(args,&args->cmpals_out, args->lines[k]);
}
if ( bcf_write1(file, args->out_hdr, args->lines[k])!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
}
else
args->keep_sum_ad = -1;
- args->out_hdr = bcf_hdr_dup(args->hdr);
+ args->out_hdr = args->hdr;
if ( args->old_rec_tag )
bcf_hdr_printf(args->out_hdr,"##INFO=<ID=%s,Number=1,Type=String,Description=\"Original variant. Format: CHR|POS|REF|ALT|USED_ALT_IDX\">",args->old_rec_tag);
args->abuf = abuf_init(args->hdr, SPLIT);
abuf_set_opt(args->abuf, bcf_hdr_t*, BCF_HDR, args->out_hdr);
if ( args->old_rec_tag )
+ {
abuf_set_opt(args->abuf, const char*, INFO_TAG, args->old_rec_tag);
+ if ( bcf_hdr_sync(args->out_hdr)!=0 ) error("bcf_hdr_sync failed\n");
+ }
abuf_set_opt(args->abuf, int, STAR_ALLELE, args->use_star_allele);
}
if ( args->gff_fname )
{
args->gff = gff_init(args->gff_fname);
- gff_set(args->gff,verbosity,1);
+ gff_set(args->gff,verbosity,args->gff_verbosity);
gff_set(args->gff,strip_chr_names,1);
gff_parse(args->gff);
args->idx_tscript = gff_get(args->gff,idx_tscript);
args->itr_tscript = regitr_init(NULL);
}
+ if ( args->filter_str )
+ args->filter = filter_init(args->hdr, args->filter_str);
+ args->filter_pass = 1;
+
+ args->out_hdr = bcf_hdr_dup(args->out_hdr);
}
static void destroy_data(args_t *args)
{
+ if ( args->filter ) filter_destroy(args->filter);
if ( args->gff )
{
gff_destroy(args->gff);
for (i=0; i<args->mtmp_lines; i++)
if ( args->tmp_lines[i] ) bcf_destroy1(args->tmp_lines[i]);
free(args->tmp_lines);
- for (i=0; i<args->malines; i++)
- bcf_destroy1(args->alines[i]);
- free(args->alines);
- for (i=0; i<args->mblines; i++)
- bcf_destroy1(args->blines[i]);
- free(args->blines);
+ for (i=0; i<args->mmrows; i++)
+ bcf_destroy1(args->mrows[i]);
+ free(args->mrows);
for (i=0; i<args->mmaps; i++)
free(args->maps[i].map);
for (i=0; i<args->ntmp_als; i++)
free(args->tmp_als[i].s);
- for (i=0; i<args->ntmp_del; i++)
- free(args->tmp_del[i].s);
+ for (i=0; i<args->ntmp_sym; i++)
+ free(args->tmp_sym[i].s);
free(args->tmp_als);
- free(args->tmp_del);
+ free(args->tmp_sym);
free(args->tmp_kstr.s);
if ( args->tmp_str )
{
{
if ( args->fai )
{
- if ( args->check_ref & CHECK_REF_FIX ) fix_ref(args, line);
+ if ( args->filter_pass && (args->check_ref & CHECK_REF_FIX) ) fix_ref(args, line);
if ( args->do_indels )
{
- int ret = realign(args, line);
+ int ret = args->filter_pass ? realign(args, line) : ERR_OK;
// exclude broken VCF lines
if ( ret==ERR_REF_MISMATCH && args->check_ref & CHECK_REF_SKIP )
}
}
- // insert into sorted buffer
- rbuf_expand0(&args->rbuf,bcf1_t*,args->rbuf.n+1,args->lines);
- int i,j;
- i = j = rbuf_append(&args->rbuf);
- if ( args->lines[i] ) bcf_destroy(args->lines[i]);
- args->lines[i] = bcf_dup(line);
- while ( rbuf_prev(&args->rbuf,&i) )
+ if ( args->filter_pass && args->atomize==SPLIT ) abuf_push(args->abuf,line);
+ while (1)
{
- if ( args->lines[i]->pos > args->lines[j]->pos ) SWAP(bcf1_t*, args->lines[i], args->lines[j]);
- j = i;
+ if ( args->filter_pass && args->atomize==SPLIT )
+ {
+ line = abuf_flush(args->abuf, 0);
+ if ( !line ) break;
+ }
+
+ // insert into sorted buffer
+ rbuf_expand0(&args->rbuf,bcf1_t*,args->rbuf.n+1,args->lines);
+ int i,j;
+ i = j = rbuf_append(&args->rbuf);
+ if ( args->lines[i] ) bcf_destroy(args->lines[i]);
+ args->lines[i] = bcf_dup(line);
+ while ( rbuf_prev(&args->rbuf,&i) )
+ {
+ if ( args->lines[i]->rid==args->lines[j]->rid )
+ {
+ bcf_unpack(args->lines[i], BCF_UN_STR);
+ bcf_unpack(args->lines[j], BCF_UN_STR);
+ if ( args->cmp_func(&args->lines[i], &args->lines[j]) > 0) SWAP(bcf1_t*, args->lines[i], args->lines[j]);
+ }
+ j = i;
+ }
+ if ( !args->filter_pass || args->atomize!=SPLIT ) break;
}
}
-static bcf1_t *next_atomized_line(args_t *args)
+// return 0 on success, 1 when done
+static int split_and_normalize(args_t *args)
{
- bcf1_t *rec = NULL;
- if ( args->atomize==SPLIT )
+ if ( !bcf_sr_next_line(args->files) ) return 1;
+
+ bcf1_t *line = bcf_sr_get_line(args->files,0);
+ args->ntotal++;
+
+ if ( args->filter )
{
- rec = abuf_flush(args->abuf, 0);
- if ( rec ) return rec;
+ args->filter_pass = filter_test(args->filter,line,NULL);
+ if ( args->filter_logic==FLT_EXCLUDE ) args->filter_pass = args->filter_pass ? 0 : 1;
+ if ( !args->filter_pass ) args->nfilter++;
}
- if ( !bcf_sr_next_line(args->files) ) return NULL;
+ if ( args->mrows_op!=MROWS_SPLIT || line->n_allele<=2 || !args->filter_pass )
+ {
+ // normal operation, no splitting
+ normalize_line(args, line);
+ return 0;
+ }
- if ( args->atomize==SPLIT )
+ // any restrictions on variant types to split?
+ if ( args->mrows_collapse!=COLLAPSE_BOTH && args->mrows_collapse!=COLLAPSE_ANY )
{
- abuf_push(args->abuf,bcf_sr_get_line(args->files,0));
- return abuf_flush(args->abuf, 0);
+ int type = args->mrows_collapse==COLLAPSE_SNPS ? VCF_SNP : VCF_INDEL;
+ if ( !(bcf_get_variant_types(line) & type) )
+ {
+ normalize_line(args, line);
+ return 0;
+ }
}
- return bcf_sr_get_line(args->files,0);
+
+ args->nsplit++;
+ split_multiallelic_to_biallelics(args, line);
+
+ int j;
+ for (j=0; j<args->ntmp_lines; j++)
+ normalize_line(args, args->tmp_lines[j]);
+
+ return 0;
}
+
static void normalize_vcf(args_t *args)
{
char wmode[8];
hts_set_opt(args->out, HTS_OPT_THREAD_POOL, args->files->p);
if (args->record_cmd_line) bcf_hdr_append_version(args->out_hdr, args->argc, args->argv, "bcftools_norm");
if ( bcf_hdr_write(args->out, args->out_hdr)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out,args->out_hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out,args->out_hdr,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
- bcf1_t *line;
- int prev_rid = -1, prev_pos = -1, prev_type = 0;
- while ( (line = next_atomized_line(args)) )
+ while (1)
{
- args->ntotal++;
- if ( args->rmdup )
- {
- int line_type = bcf_get_variant_types(line);
- if ( prev_rid>=0 && prev_rid==line->rid && prev_pos==line->pos )
- {
- if ( args->rmdup & BCF_SR_PAIR_ANY ) continue; // rmdup by position only
- if ( args->rmdup & BCF_SR_PAIR_SNPS && line_type&(VCF_SNP|VCF_MNP) && prev_type&(VCF_SNP|VCF_MNP) ) continue;
- if ( args->rmdup & BCF_SR_PAIR_INDELS && line_type&(VCF_INDEL) && prev_type&(VCF_INDEL) ) continue;
- if ( args->rmdup & BCF_SR_PAIR_EXACT && cmpals_match(&args->cmpals_in, line) ) continue;
- }
- else
- {
- prev_rid = line->rid;
- prev_pos = line->pos;
- prev_type = 0;
- if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_reset(&args->cmpals_in);
- }
- prev_type |= line_type;
- if ( args->rmdup & BCF_SR_PAIR_EXACT ) cmpals_add(&args->cmpals_in, line);
- }
-
- // still on the same chromosome?
- int i,j,ilast = rbuf_last(&args->rbuf);
- if ( ilast>=0 && line->rid != args->lines[ilast]->rid ) flush_buffer(args, args->out, args->rbuf.n); // new chromosome
-
- int split = 0;
- if ( args->mrows_op==MROWS_SPLIT )
+ // buffer lines until a different position is reached
+ int done = 0;
+ while (1)
{
- split = 1;
- if ( args->mrows_collapse!=COLLAPSE_BOTH && args->mrows_collapse!=COLLAPSE_ANY )
- {
- if ( !(bcf_get_variant_types(line) & args->mrows_collapse) ) split = 0;
- }
- if ( split && line->n_allele>2 )
- {
- args->nsplit++;
- split_multiallelic_to_biallelics(args, line);
- for (j=0; j<args->ntmp_lines; j++)
- normalize_line(args, args->tmp_lines[j]);
- }
- else
- split = 0;
+ done = split_and_normalize(args);
+ if ( done ) break; // no more lines available
+ int i = args->rbuf.f;
+ int j = rbuf_last(&args->rbuf);
+ if ( args->lines[i]->rid != args->lines[j]->rid ) break;
+ if ( args->lines[i]->pos != args->lines[j]->pos ) break;
}
- if ( !split )
- normalize_line(args, line);
+ if ( done ) break;
// find out how many sites to flush
- ilast = rbuf_last(&args->rbuf);
- j = 0;
+ int ifst = args->rbuf.f;
+ int ilast = rbuf_last(&args->rbuf);
+ int i, j = 0;
for (i=-1; rbuf_next(&args->rbuf,&i); )
{
+ if ( args->lines[ifst]->rid != args->lines[ilast]->rid )
+ {
+ // there are two chromosomes in the buffer, count how many are on the first chromosome
+ if ( args->lines[ifst]->rid != args->lines[i]->rid ) break;
+ j++;
+ continue;
+ }
+ // there is just one chromosome, flush only lines that are unlikely to change order on
+ // realigning (the buf_win constant)
if ( args->lines[ilast]->pos - args->lines[i]->pos < args->buf_win ) break;
j++;
}
}
if ( hts_close(args->out)!=0 ) error("[%s] Error: close failed .. %s\n", __func__,args->output_fname);
- fprintf(bcftools_stderr,"Lines total/split/realigned/skipped:\t%d/%d/%d/%d\n", args->ntotal,args->nsplit,args->nchanged,args->nskipped);
+ fprintf(bcftools_stderr,"Lines total/split/joined/realigned/removed/skipped:\t%d/%d/%d/%d/%d/%d\n",
+ args->ntotal,args->nsplit,args->njoined,args->nchanged,args->nskipped,args->nfilter);
if ( args->check_ref & CHECK_REF_FIX )
fprintf(bcftools_stderr,"REF/ALT total/modified/added: \t%d/%d/%d\n", args->nref.tot,args->nref.swap,args->nref.set);
}
fprintf(bcftools_stderr, " -c, --check-ref e|w|x|s Check REF alleles and exit (e), warn (w), exclude (x), or set (s) bad sites [e]\n");
fprintf(bcftools_stderr, " -D, --remove-duplicates Remove duplicate lines of the same type.\n");
fprintf(bcftools_stderr, " -d, --rm-dup TYPE Remove duplicate snps|indels|both|all|exact\n");
+ fprintf(bcftools_stderr, " -e, --exclude EXPR Do not normalize records for which the expression is true (see man page for details)\n");
fprintf(bcftools_stderr, " -f, --fasta-ref FILE Reference sequence\n");
fprintf(bcftools_stderr, " --force Try to proceed even if malformed tags are encountered. Experimental, use at your own risk\n");
fprintf(bcftools_stderr, " -g, --gff-annot FILE Follow HGVS 3'rule and right-align variants in transcripts on the forward strand\n");
+ fprintf(bcftools_stderr, " -i, --include EXPR Normalize only records for which the expression is true (see man page for details)\n");
fprintf(bcftools_stderr, " --keep-sum TAG,.. Keep vector sum constant when splitting multiallelics (see github issue #360)\n");
fprintf(bcftools_stderr, " -m, --multiallelics -|+TYPE Split multiallelics (-) or join biallelics (+), type: snps|indels|both|any [both]\n");
fprintf(bcftools_stderr, " --multi-overlaps 0|. Fill in the reference (0) or missing (.) allele when splitting multiallelics [0]\n");
fprintf(bcftools_stderr, " -R, --regions-file FILE Restrict to regions listed in a file\n");
fprintf(bcftools_stderr, " --regions-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [1]\n");
fprintf(bcftools_stderr, " -s, --strict-filter When merging (-m+), merged site is PASS only if all sites being merged PASS\n");
+ fprintf(bcftools_stderr, " -S, --sort METHOD Sort order: chr_pos,lex [chr_pos]\n");
fprintf(bcftools_stderr, " -t, --targets REGION Similar to -r but streams rather than index-jumps\n");
fprintf(bcftools_stderr, " -T, --targets-file FILE Similar to -R but streams rather than index-jumps\n");
fprintf(bcftools_stderr, " --targets-overlap 0|1|2 Include if POS in the region (0), record overlaps (1), variant overlaps (2) [0]\n");
- fprintf(bcftools_stderr, " --threads INT Use multithreading with <int> worker threads [0]\n");
+ fprintf(bcftools_stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
+ fprintf(bcftools_stderr, " -v, --verbose INT Verbosity level (0-2) of GFF parsing [1]\n");
fprintf(bcftools_stderr, " -w, --site-win INT Buffer for sorting lines which changed position during realignment [1000]\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Examples:\n");
fprintf(bcftools_stderr, " # normalize and left-align indels\n");
int region_is_file = 0;
int targets_is_file = 0;
args->use_star_allele = 1;
+ args->gff_verbosity = 1;
int regions_overlap = 1;
int targets_overlap = 0;
+ args->cmp_func = cmp_bcf_pos;
static struct option loptions[] =
{
{"force",no_argument,NULL,7},
{"atomize",no_argument,NULL,'a'},
{"atom-overlaps",required_argument,NULL,11},
+ {"include",required_argument,NULL,'i'},
+ {"exclude",required_argument,NULL,'e'},
{"old-rec-tag",required_argument,NULL,12},
{"keep-sum",required_argument,NULL,10},
{"fasta-ref",required_argument,NULL,'f'},
+ {"sort",required_argument,NULL,'S'},
{"gff-annot",required_argument,NULL,'g'},
{"right-align",no_argument,NULL,15}, // undocumented, only for debugging
{"do-not-normalize",no_argument,NULL,'N'},
{"check-ref",required_argument,NULL,'c'},
{"strict-filter",no_argument,NULL,'s'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,14},
+ {"write-index",optional_argument,NULL,'W'},
+ {"verbose",required_argument,NULL,'v'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "hr:R:f:w:Dd:o:O:c:m:t:T:sNag:",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hr:R:f:w:Dd:o:O:c:m:t:T:sNag:W::v:S:i:e:",loptions,NULL)) >= 0) {
switch (c) {
case 10:
// possibly generalize this also to INFO/AD and other tags
args->keep_sum_ad = 1; // this will be set to the header id or -1 in init_data
break;
case 'g': args->gff_fname = optarg; break;
+ case 'v':
+ args->gff_verbosity = atoi(optarg);
+ if ( args->gff_verbosity<0 || args->gff_verbosity>2 ) error("Error: expected integer 0-2 with -v, --verbose\n");
+ break;
case 'a': args->atomize = SPLIT; break;
+ case 'e':
+ if ( args->filter_str ) error("Error: only one -i or -e expression can be given, and they cannot be combined\n");
+ args->filter_str = optarg; args->filter_logic |= FLT_EXCLUDE; break;
+ case 'i':
+ if ( args->filter_str ) error("Error: only one -i or -e expression can be given, and they cannot be combined\n");
+ args->filter_str = optarg; args->filter_logic |= FLT_INCLUDE; break;
+ case 'S':
+ if ( !strcasecmp(optarg,"pos") ) args->cmp_func = cmp_bcf_pos;
+ else if ( !strcasecmp(optarg,"lex") ) args->cmp_func = cmp_bcf_pos_ref_alt;
+ else error("Error: the sort order --sort %s is not recognised\n",optarg);
+ break;
case 11 :
if ( optarg[0]=='*' ) args->use_star_allele = 1;
else if ( optarg[0]=='.' ) args->use_star_allele = 0;
else if ( optarg[0]=='.' ) args->ma_use_ref_allele = 0;
else error("Invalid argument to --multi-overlaps\n");
break;
- case 14 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 15 : args->right_align = 1; break;
case 'N': args->do_indels = 0; break;
case 'd':
if ( bcf_sr_set_threads(args->files, args->n_threads)<0 ) error("Failed to create threads\n");
if ( !bcf_sr_add_reader(args->files, fname) ) error("Failed to read from %s: %s\n", !strcmp("-",fname)?"standard input":fname,bcf_sr_strerror(args->files->errnum));
- if ( args->mrows_op&MROWS_SPLIT && args->rmdup ) error("Cannot combine -D and -m-\n");
init_data(args);
normalize_vcf(args);
destroy_data(args);
* success or non-zero value on error.
*
* int init(int argc, char **argv, bcf_hdr_t *in_hdr, bcf_hdr_t *out_hdr)
- * - called once at startup, allows to initialize local variables.
+ * - called once at startup, it initializes local variables.
* Return 1 to suppress normal VCF/BCF header output, -1 on critical
* errors, 0 otherwise.
*
if ( args->out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->output_fname, strerror(errno));
if ( args->n_threads ) hts_set_threads(args->out_fh, args->n_threads);
if ( bcf_hdr_write(args->out_fh, args->hdr_out)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->hdr_out,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr_out,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
}
}
fprintf(stderr, " -l, --list-plugins List available plugins. See BCFTOOLS_PLUGINS environment variable and man page for details\n");
fprintf(stderr, " -v, --verbose Print verbose information, -vv increases verbosity\n");
fprintf(stderr, " -V, --version Print version string and exit\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
exit(1);
}
{"targets-file",required_argument,NULL,'T'},
{"targets-overlap",required_argument,NULL,2},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "h?o:O:r:R:t:T:li:e:vV",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h?o:O:r:R:t:T:li:e:vVW::",loptions,NULL)) >= 0)
{
switch (c) {
case 'V': version_only = 1; break;
break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
- case 10 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case '?':
case 'h': usage_only = 1; break;
default: error("Unknown argument: %s\n", optarg);
* success or non-zero value on error.
*
* int init(int argc, char **argv, bcf_hdr_t *in_hdr, bcf_hdr_t *out_hdr)
- * - called once at startup, allows to initialize local variables.
+ * - called once at startup, it initializes local variables.
* Return 1 to suppress normal VCF/BCF header output, -1 on critical
* errors, 0 otherwise.
*
if ( args->out_fh == NULL ) error("Can't write to \"%s\": %s\n", args->output_fname, strerror(errno));
if ( args->n_threads ) hts_set_threads(args->out_fh, args->n_threads);
if ( bcf_hdr_write(args->out_fh, args->hdr_out)!=0 ) error("[%s] Error: cannot write to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(args->out_fh,args->hdr_out,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( init_index2(args->out_fh,args->hdr_out,args->output_fname,
+ &args->index_fn, args->write_index)<0 )
+ error("Error: failed to initialise index for %s\n",args->output_fname);
}
}
fprintf(bcftools_stderr, " -l, --list-plugins List available plugins. See BCFTOOLS_PLUGINS environment variable and man page for details\n");
fprintf(bcftools_stderr, " -v, --verbose Print verbose information, -vv increases verbosity\n");
fprintf(bcftools_stderr, " -V, --version Print version string and exit\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
{"targets-file",required_argument,NULL,'T'},
{"targets-overlap",required_argument,NULL,2},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "h?o:O:r:R:t:T:li:e:vV",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "h?o:O:r:R:t:T:li:e:vVW::",loptions,NULL)) >= 0)
{
switch (c) {
case 'V': version_only = 1; break;
break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
- case 10 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case '?':
case 'h': usage_only = 1; break;
default: error("Unknown argument: %s\n", optarg);
bcf_hdr_t *header;
int sample_is_file;
char **argv, *format_str, *sample_list, *targets_list, *regions_list, *vcf_list, *fn_out;
- int argc, list_columns, print_header, allow_undef_tags, force_samples;
+ char *print_filtered;
+ int argc, list_columns, print_header, allow_undef_tags, force_samples, force_newline;
FILE *out;
}
args_t;
smpl_ilist_destroy(ilist);
}
args->convert = convert_init(args->header, samples, nsamples, args->format_str);
- convert_set_option(args->convert, force_newline, 1);
+ if ( args->force_newline ) convert_set_option(args->convert, force_newline, 1);
convert_set_option(args->convert, subset_samples, &args->smpl_pass);
if ( args->allow_undef_tags ) convert_set_option(args->convert, allow_undef_tags, 1);
+ if ( args->print_header>1 ) convert_set_option(args->convert, no_hdr_indices, 1);
free(samples);
int max_unpack = convert_max_unpack(args->convert);
max_unpack |= filter_max_unpack(args->filter);
}
args->files->max_unpack = max_unpack;
+ if ( !args->filter || args->print_filtered || !(filter_max_unpack(args->filter) & BCF_UN_FMT) )
+ convert_set_option(args->convert, header_samples, 1);
+ if ( args->print_filtered ) convert_set_option(args->convert, print_filtered, args->print_filtered);
}
static void destroy_data(args_t *args)
fprintf(stderr, "Options:\n");
fprintf(stderr, " -e, --exclude EXPR Exclude sites for which the expression is true (see man page for details)\n");
fprintf(stderr, " --force-samples Only warn about unknown subset samples\n");
+ fprintf(stderr, " -F, --print-filtered STR Output STR for samples failing the -i/-e filtering expression\n");
fprintf(stderr, " -f, --format STRING See man page for details\n");
- fprintf(stderr, " -H, --print-header Print header\n");
+ fprintf(stderr, " -H, --print-header Print header, -HH to omit column indices\n");
fprintf(stderr, " -i, --include EXPR Select sites for which the expression is true (see man page for details)\n");
fprintf(stderr, " -l, --list-samples Print the list of samples and exit\n");
+ fprintf(stderr, " -N, --disable-automatic-newline Disable automatic addition of newline character when not present\n");
fprintf(stderr, " -o, --output FILE Output file name [stdout]\n");
fprintf(stderr, " -r, --regions REGION Restrict to comma-separated list of regions\n");
fprintf(stderr, " -R, --regions-file FILE Restrict to regions listed in a file\n");
fprintf(stderr, "\n");
fprintf(stderr, "Examples:\n");
fprintf(stderr, "\tbcftools query -f '%%CHROM\\t%%POS\\t%%REF\\t%%ALT[\\t%%SAMPLE=%%GT]\\n' file.vcf.gz\n");
+ fprintf(stderr, "\t# For more examples see http://samtools.github.io/bcftools/bcftools.html#query\n");
fprintf(stderr, "\n");
exit(1);
}
int c, collapse = 0;
args_t *args = (args_t*) calloc(1,sizeof(args_t));
args->argc = argc; args->argv = argv;
+ args->force_newline = 1;
int regions_is_file = 0, targets_is_file = 0;
int regions_overlap = 1;
int targets_overlap = 0;
{
{"help",0,0,'h'},
{"list-samples",0,0,'l'},
+ {"disable-automatic-newline",required_argument,NULL,'N'},
{"include",1,0,'i'},
{"exclude",1,0,'e'},
+ {"print-filtered",1,0,'F'},
{"format",1,0,'f'},
{"force-samples",0,0,3},
{"output-file",1,0,'o'},
{"allow-undef-tags",0,0,'u'},
{0,0,0,0}
};
- while ((c = getopt_long(argc, argv, "hlr:R:f:a:s:S:Ht:T:c:v:i:e:o:u",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hlr:R:F:f:a:s:S:Ht:T:c:v:i:e:o:uN",loptions,NULL)) >= 0) {
switch (c) {
case 'o': args->fn_out = optarg; break;
+ case 'F': args->print_filtered = optarg; break;
case 'f': args->format_str = strdup(optarg); break;
- case 'H': args->print_header = 1; break;
+ case 'N': args->force_newline = 0; break;
+ case 'H': args->print_header++; break;
case 'v': args->vcf_list = optarg; break;
case 'c':
error("The --collapse option is obsolete, pipe through `bcftools norm -c` instead.\n");
bcf_hdr_t *header;
int sample_is_file;
char **argv, *format_str, *sample_list, *targets_list, *regions_list, *vcf_list, *fn_out;
- int argc, list_columns, print_header, allow_undef_tags, force_samples;
+ char *print_filtered;
+ int argc, list_columns, print_header, allow_undef_tags, force_samples, force_newline;
FILE *out;
}
args_t;
smpl_ilist_destroy(ilist);
}
args->convert = convert_init(args->header, samples, nsamples, args->format_str);
- convert_set_option(args->convert, force_newline, 1);
+ if ( args->force_newline ) convert_set_option(args->convert, force_newline, 1);
convert_set_option(args->convert, subset_samples, &args->smpl_pass);
if ( args->allow_undef_tags ) convert_set_option(args->convert, allow_undef_tags, 1);
+ if ( args->print_header>1 ) convert_set_option(args->convert, no_hdr_indices, 1);
free(samples);
int max_unpack = convert_max_unpack(args->convert);
max_unpack |= filter_max_unpack(args->filter);
}
args->files->max_unpack = max_unpack;
+ if ( !args->filter || args->print_filtered || !(filter_max_unpack(args->filter) & BCF_UN_FMT) )
+ convert_set_option(args->convert, header_samples, 1);
+ if ( args->print_filtered ) convert_set_option(args->convert, print_filtered, args->print_filtered);
}
static void destroy_data(args_t *args)
fprintf(bcftools_stderr, "Options:\n");
fprintf(bcftools_stderr, " -e, --exclude EXPR Exclude sites for which the expression is true (see man page for details)\n");
fprintf(bcftools_stderr, " --force-samples Only warn about unknown subset samples\n");
+ fprintf(bcftools_stderr, " -F, --print-filtered STR Output STR for samples failing the -i/-e filtering expression\n");
fprintf(bcftools_stderr, " -f, --format STRING See man page for details\n");
- fprintf(bcftools_stderr, " -H, --print-header Print header\n");
+ fprintf(bcftools_stderr, " -H, --print-header Print header, -HH to omit column indices\n");
fprintf(bcftools_stderr, " -i, --include EXPR Select sites for which the expression is true (see man page for details)\n");
fprintf(bcftools_stderr, " -l, --list-samples Print the list of samples and exit\n");
+ fprintf(bcftools_stderr, " -N, --disable-automatic-newline Disable automatic addition of newline character when not present\n");
fprintf(bcftools_stderr, " -o, --output FILE Output file name [bcftools_stdout]\n");
fprintf(bcftools_stderr, " -r, --regions REGION Restrict to comma-separated list of regions\n");
fprintf(bcftools_stderr, " -R, --regions-file FILE Restrict to regions listed in a file\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Examples:\n");
fprintf(bcftools_stderr, "\tbcftools query -f '%%CHROM\\t%%POS\\t%%REF\\t%%ALT[\\t%%SAMPLE=%%GT]\\n' file.vcf.gz\n");
+ fprintf(bcftools_stderr, "\t# For more examples see http://samtools.github.io/bcftools/bcftools.html#query\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
int c, collapse = 0;
args_t *args = (args_t*) calloc(1,sizeof(args_t));
args->argc = argc; args->argv = argv;
+ args->force_newline = 1;
int regions_is_file = 0, targets_is_file = 0;
int regions_overlap = 1;
int targets_overlap = 0;
{
{"help",0,0,'h'},
{"list-samples",0,0,'l'},
+ {"disable-automatic-newline",required_argument,NULL,'N'},
{"include",1,0,'i'},
{"exclude",1,0,'e'},
+ {"print-filtered",1,0,'F'},
{"format",1,0,'f'},
{"force-samples",0,0,3},
{"output-file",1,0,'o'},
{"allow-undef-tags",0,0,'u'},
{0,0,0,0}
};
- while ((c = getopt_long(argc, argv, "hlr:R:f:a:s:S:Ht:T:c:v:i:e:o:u",loptions,NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "hlr:R:F:f:a:s:S:Ht:T:c:v:i:e:o:uN",loptions,NULL)) >= 0) {
switch (c) {
case 'o': args->fn_out = optarg; break;
+ case 'F': args->print_filtered = optarg; break;
case 'f': args->format_str = strdup(optarg); break;
- case 'H': args->print_header = 1; break;
+ case 'N': args->force_newline = 0; break;
+ case 'H': args->print_header++; break;
case 'v': args->vcf_list = optarg; break;
case 'c':
error("The --collapse option is obsolete, pipe through `bcftools norm -c` instead.\n");
{
if ( *end!=',') error("Could not parse: --buffer-size %s\n", args->buffer_size);
args->nbuf_olap = strtol(end+1,&end,10);
- if ( *end || args->nbuf_olap<0 ) error("Could not parse: --bufer-size %s\n", args->buffer_size);
+ if ( *end || args->nbuf_olap<0 ) error("Could not parse: --buffer-size %s\n", args->buffer_size);
}
if ( tmp<0 )
args->nbuf_max = fabs(tmp)*1e6/(4+8*2)/args->roh_smpl->n;
{
if ( *end!=',') error("Could not parse: --buffer-size %s\n", args->buffer_size);
args->nbuf_olap = strtol(end+1,&end,10);
- if ( *end || args->nbuf_olap<0 ) error("Could not parse: --bufer-size %s\n", args->buffer_size);
+ if ( *end || args->nbuf_olap<0 ) error("Could not parse: --buffer-size %s\n", args->buffer_size);
}
if ( tmp<0 )
args->nbuf_max = fabs(tmp)*1e6/(4+8*2)/args->roh_smpl->n;
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/hts_os.h>
+#include <htslib/hts_defs.h>
#include <inttypes.h>
#include "bcftools.h"
args_t;
static void usage(void);
-FILE *open_file(char **fname, const char *mode, const char *fmt, ...);
-void mkdir_p(const char *fmt, ...);
+FILE *open_file(char **fname, const char *mode, const char *fmt, ...) HTS_FORMAT(HTS_PRINTF_FMT, 3, 4);
-char *msprintf(const char *fmt, ...)
+char * HTS_FORMAT(HTS_PRINTF_FMT, 1, 2) msprintf(const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
#include <htslib/hts_os.h>
+#include <htslib/hts_defs.h>
#include <inttypes.h>
#include "bcftools.h"
args_t;
static void usage(void);
-FILE *open_file(char **fname, const char *mode, const char *fmt, ...);
-void mkdir_p(const char *fmt, ...);
+FILE *open_file(char **fname, const char *mode, const char *fmt, ...) HTS_FORMAT(HTS_PRINTF_FMT, 3, 4);
-char *msprintf(const char *fmt, ...)
+char * HTS_FORMAT(HTS_PRINTF_FMT, 1, 2) msprintf(const char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
/* vcfsort.c -- sort subcommand
- Copyright (C) 2017-2023 Genome Research Ltd.
+ Copyright (C) 2017-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/vcf.h>
#include <htslib/kstring.h>
#include <htslib/hts_os.h>
+#include <htslib/hts_defs.h>
+#include <htslib/bgzf.h>
#include "kheap.h"
#include "bcftools.h"
+#define MAX_TMP_FILES_PER_LAYER 32
+#define MERGE_LAYERS 12
+#define MAX_TMP_FILES (MAX_TMP_FILES_PER_LAYER * MERGE_LAYERS)
+
typedef struct
{
char *fname;
htsFile *fh;
+ BGZF *bgz;
+ size_t idx;
bcf1_t *rec;
+ int is_merged;
}
blk_t;
+typedef struct
+{
+ size_t len;
+ hts_pos_t pos;
+ int rid;
+ float qual;
+ uint8_t data[];
+}
+packed_bcf_t;
+
typedef struct _args_t
{
bcf_hdr_t *hdr;
char **argv, *fname, *output_fname, *tmp_dir;
int argc, output_type, clevel;
size_t max_mem, mem;
- bcf1_t **buf;
+ packed_bcf_t **buf;
uint8_t *mem_block;
- size_t nbuf, mbuf, nblk;
- blk_t *blk;
- char *index_fn;
+
+ size_t nbuf, mbuf, nblk, tmp_count;
+ blk_t blk[MAX_TMP_FILES];
+ uint32_t tmp_layers[MERGE_LAYERS];
int write_index;
}
args_t;
{
int i;
fprintf(stderr,"Cleaning\n");
- for (i=0; i<args->nblk; i++)
+ for (i=0; i<MAX_TMP_FILES; i++)
{
- blk_t *blk = args->blk + i;
+ blk_t *blk = &args->blk[i];
if ( blk->fname )
{
unlink(blk->fname);
}
rmdir(args->tmp_dir);
}
-void clean_files_and_throw(args_t *args, const char *format, ...)
+void HTS_FORMAT(HTS_PRINTF_FMT, 2, 3) HTS_NORETURN
+clean_files_and_throw(args_t *args, const char *format, ...)
{
va_list ap;
va_start(ap, format);
}
int cmp_bcf_pos(const void *aptr, const void *bptr)
+{
+ bcf1_t *a = *((bcf1_t**)aptr);
+ bcf1_t *b = *((bcf1_t**)bptr);
+ if ( a->rid < b->rid ) return -1;
+ if ( a->rid > b->rid ) return 1;
+ if ( a->pos < b->pos ) return -1;
+ if ( a->pos > b->pos ) return 1;
+ return 0;
+}
+int cmp_bcf_pos_ref_alt(const void *aptr, const void *bptr)
{
bcf1_t *a = *((bcf1_t**)aptr);
bcf1_t *b = *((bcf1_t**)bptr);
return 0;
}
-void buf_flush(args_t *args)
+static int cmp_packed_bcf_pos_ref_alt(const void *aptr, const void *bptr)
{
- if ( !args->nbuf ) return;
+ packed_bcf_t *a = *(packed_bcf_t **) aptr;
+ packed_bcf_t *b = *(packed_bcf_t **) bptr;
- qsort(args->buf, args->nbuf, sizeof(*args->buf), cmp_bcf_pos);
+ if ( a->rid < b->rid ) return -1;
+ if ( a->rid > b->rid ) return 1;
+ if ( a->pos < b->pos ) return -1;
+ if ( a->pos > b->pos ) return 1;
+
+ // Sort lexicographically by ref,alt. These are stored tab-separated
+ // as the first item in packed_bcf_t::data
+ return strcmp((char *) a->data, (char *) b->data);
+}
- args->nblk++;
- args->blk = (blk_t*) realloc(args->blk, sizeof(blk_t)*args->nblk);
- if ( !args->blk ) error("Error: could not allocate %zu bytes of memory, try reducing --max-mem\n",sizeof(blk_t)*args->nblk);
- blk_t *blk = args->blk + args->nblk - 1;
+static int cmp_packed_bcf_pos_ref_alt_stable(const void *aptr, const void *bptr)
+{
+ // cmp_bcf_pos_ref_alt() with tie-breaker to make qsort stable
+ int res = cmp_packed_bcf_pos_ref_alt(aptr, bptr);
+ if (res != 0) return res;
+
+ // Got a tie - use the position in the original input to break it.
+ // As everything is read into a big memory buffer, for most records
+ // we can just compare the pointers directly. The exception is
+ // any record that didn't quite fit in the memory buffer, causing it to be
+ // flushed. Those are flagged by setting packed_bcf_t::len = SIZE_MAX, and
+ // as they were the last record in the segment, they should always sort
+ // after unflagged records.
+
+ packed_bcf_t *a = *(packed_bcf_t **) aptr;
+ packed_bcf_t *b = *(packed_bcf_t **) bptr;
+
+ if (a->len == SIZE_MAX) return 1;
+ if (b->len == SIZE_MAX) return -1;
+
+ return a < b ? -1 : 1;
+}
+
+static uint8_t *pack_unsigned(uint8_t *data, uint64_t val)
+{
+ do {
+ *data++ = (val & 0x7f) | ((val > 0x7f) ? 0x80 : 0);
+ val >>= 7;
+ } while (val > 0);
+ return data;
+}
+
+static uint8_t *pack_hts_pos(uint8_t *data, hts_pos_t val)
+{
+ uint64_t sign = val < 0;
+ uint64_t v = val < 0 ? -(val + 1) : val;
+ v = v << 1 | sign;
+ return pack_unsigned(data, v);
+}
+
+static uint8_t *pack_bcf_data(packed_bcf_t *dest, const bcf1_t *src,
+ int outside_buffer)
+{
+ uint32_t i;
+ uint8_t *data = dest->data;
+ uint8_t *start = dest->data;
+ dest->pos = src->pos;
+ dest->rid = src->rid;
+ dest->qual = src->qual;
+
+ // Copy in alleles, for the comparison function
+ for (i = 0; i < src->n_allele; i++)
+ {
+ size_t l = strlen(src->d.allele[i]);
+ if (i > 0) *data++ = '\t';
+ memcpy(data, src->d.allele[i], l);
+ data += l;
+ }
+ *data++ = '\0';
+
+ if (outside_buffer)
+ {
+ dest->len = SIZE_MAX;
+ memcpy(data, &src, sizeof(src));
+ data += sizeof(src);
+ return data;
+ }
+
+ data = pack_hts_pos(data, src->rlen);
+ data = pack_unsigned(data, src->n_info);
+ data = pack_unsigned(data, src->n_allele);
+ data = pack_unsigned(data, src->n_fmt);
+ data = pack_unsigned(data, src->n_sample);
+ data = pack_unsigned(data, src->shared.l);
+ data = pack_unsigned(data, src->indiv.l);
+ if (src->shared.l)
+ memcpy(data, src->shared.s, src->shared.l);
+ data += src->shared.l;
+ if (src->indiv.l)
+ memcpy(data, src->indiv.s, src->indiv.l);
+ data += src->indiv.l;
+ dest->len = data - start;
+ return data;
+}
+
+static int write_packed_bcf(BGZF *fp, packed_bcf_t *src)
+{
+ // Write pos, rid, qual
+ size_t len = src->data - (uint8_t *) &src->pos;
+ if (bgzf_write_small(fp, &src->pos, len) < len)
+ return -1;
+
+ // Skip the copy of the alleles
+ size_t skip = strlen((char *) src->data) + 1;
+
+ // Write everything else
+ if (src->len < SIZE_MAX)
+ {
+ // In main memory block
+ len = src->len - skip;
+ if (bgzf_write_small(fp, src->data + skip, len) < len)
+ return -1;
+ }
+ else
+ {
+ // Record didn't fit in the main block. To minimize the
+ // overflow, its packed_bcf_t data will be imcomplete. A pointer to
+ // its bcf1_t struct will have been placed after the allele data
+ // so we can finish the packing job and write it in the same format
+ // the rest of the data
+ bcf1_t *rec;
+ uint8_t tmp[100], *data = tmp;
+ memcpy(&rec, src->data + skip, sizeof(rec));
+
+ data = pack_hts_pos(data, rec->rlen);
+ data = pack_unsigned(data, rec->n_info);
+ data = pack_unsigned(data, rec->n_allele);
+ data = pack_unsigned(data, rec->n_fmt);
+ data = pack_unsigned(data, rec->n_sample);
+ data = pack_unsigned(data, rec->shared.l);
+ data = pack_unsigned(data, rec->indiv.l);
+ if (bgzf_write_small(fp, tmp, data - tmp) < data - tmp)
+ return -1;
+ if (rec->shared.l > 0 &&
+ bgzf_write_small(fp, rec->shared.s, rec->shared.l) < rec->shared.l)
+ return -1;
+ if (rec->indiv.l > 0 &&
+ bgzf_write_small(fp, rec->indiv.s, rec->indiv.l) < rec->indiv.l)
+ return -1;
+ }
+
+ return 0;
+}
+
+static uint64_t unpack_unsigned(BGZF *fp, int *err)
+{
+ uint8_t data;
+ uint64_t val = 0;
+ uint32_t i = 0;
+
+ if (bgzf_read_small(fp, &data, sizeof(data)) <= 0)
+ goto short_read;
+
+ while (data & 0x80)
+ {
+ val |= (uint64_t)(data & 0x7f) << i;
+ i += 7;
+ if (bgzf_read_small(fp, &data, sizeof(data)) <= 0)
+ goto short_read;
+ }
+ val |= (uint64_t)data << i;
+ return val;
+
+ short_read:
+ *err = 1;
+ return 0;
+}
+
+static hts_pos_t unpack_hts_pos(BGZF *fp, int *err)
+{
+ uint64_t v = unpack_unsigned(fp, err);
+
+ if ((v & 1) == 0)
+ return (hts_pos_t)(v >> 1);
+ else
+ return -(hts_pos_t)(v >> 1) - 1;
+}
+
+static int read_packed_bcf(BGZF *fp, bcf1_t *dest)
+{
+ int err = 0;
+ packed_bcf_t tmp;
+ size_t len = tmp.data - (uint8_t *) &tmp.pos;
+
+ bcf_clear(dest);
+ ssize_t got = bgzf_read_small(fp, &tmp.pos, len);
+ if (got == 0)
+ return -1; // EOF
+ if (got < len)
+ return -2; // Error or short read
+ dest->pos = tmp.pos;
+ dest->rid = tmp.rid;
+ dest->qual = tmp.qual;
+ dest->rlen = unpack_hts_pos(fp, &err);
+ dest->n_info = unpack_unsigned(fp, &err);
+ dest->n_allele = unpack_unsigned(fp, &err);
+ dest->n_fmt = unpack_unsigned(fp, &err);
+ dest->n_sample = unpack_unsigned(fp, &err);
+ len = unpack_unsigned(fp, &err);
+ if (ks_resize(&dest->shared, len) != 0)
+ return -2;
+ dest->shared.l = len;
+ len = unpack_unsigned(fp, &err);
+ if (ks_resize(&dest->indiv, len) != 0)
+ return -2;
+ dest->indiv.l = len;
+ err |= bgzf_read_small(fp, dest->shared.s, dest->shared.l) < dest->shared.l;
+ err |= bgzf_read_small(fp, dest->indiv.s, dest->indiv.l) < dest->indiv.l;
+ return err == 0 ? 0 : -2;
+}
+void open_tmp_file(args_t *args, blk_t *blk, int is_merged)
+{
kstring_t str = {0,0,0};
- ksprintf(&str, "%s/%05d.bcf", args->tmp_dir, (int)args->nblk);
- blk->fname = str.s;
- blk->rec = NULL;
- blk->fh = NULL;
+ int tries = 1000;
- htsFile *fh = hts_open(blk->fname, "wbu");
- if ( fh == NULL ) clean_files_and_throw(args, "Cannot write %s: %s\n", blk->fname, strerror(errno));
- if ( bcf_hdr_write(fh, args->hdr)!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,blk->fname);
+ blk->fh = NULL;
+ blk->bgz = NULL;
+
+ do {
+ if (ksprintf(ks_clear(&str), "%s/%05zd%s",
+ args->tmp_dir, args->tmp_count++,
+ is_merged ? ".bcf" : "") < 0) {
+ clean_files_and_throw(args, "%s", strerror(errno));
+ }
+
+ if (is_merged)
+ blk->fh = hts_open(str.s, "wbx1");
+ else
+ blk->bgz = bgzf_open(str.s, "wx1");
+ if ( blk->fh == NULL && blk->bgz == NULL && (errno != EEXIST || --tries <= 0)) {
+ clean_files_and_throw(args, "Cannot write %s: %s\n",
+ str.s, strerror(errno));
+ }
+ } while (blk->fh == NULL && blk->bgz == NULL);
+
+ blk->fname = ks_release(&str);
+ blk->idx = args->tmp_count - 1;
+}
+
+void do_partial_merge(args_t *args);
+
+void buf_flush(args_t *args, bcf1_t *last_rec)
+{
+ if ( !args->nbuf ) return;
+
+ qsort(args->buf, args->nbuf, sizeof(*args->buf), cmp_packed_bcf_pos_ref_alt_stable);
+
+ if (args->tmp_layers[0] >= MAX_TMP_FILES_PER_LAYER)
+ do_partial_merge(args);
+
+ assert(args->nblk < MAX_TMP_FILES);
+ blk_t *blk = &args->blk[args->nblk];
+ blk->is_merged = 0;
+ args->nblk++;
+ args->tmp_layers[0]++;
+ assert(blk->fname == NULL && blk->fh == NULL && blk->bgz == NULL);
+
+ open_tmp_file(args, blk, 0);
int i;
for (i=0; i<args->nbuf; i++)
{
- if ( bcf_write(fh, args->hdr, args->buf[i])!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,blk->fname);
+ if ( write_packed_bcf(blk->bgz, args->buf[i])!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,blk->fname);
}
- if ( hts_close(fh)!=0 ) clean_files_and_throw(args, "[%s] Error: close failed .. %s\n", __func__,blk->fname);
+
+ if ( bgzf_close(blk->bgz)!=0 ) clean_files_and_throw(args, "[%s] Error: close failed .. %s\n", __func__,blk->fname);
+ blk->bgz = NULL;
args->nbuf = 0;
args->mem = 0;
return (uint8_t*)(((size_t)ptr + 8 - 1) & ~((size_t)(8 - 1)));
}
+#define varint_size(X) ((sizeof(X) * 8 + 7) / 7) // worst case
+
void buf_push(args_t *args, bcf1_t *rec)
{
- size_t delta = sizeof(bcf1_t) + rec->shared.l + rec->indiv.l + rec->unpack_size[0] + rec->unpack_size[1]
- + sizeof(*rec->d.allele)*rec->d.m_allele
- + sizeof(bcf1_t*) // args->buf
+ size_t delta = sizeof(rec->pos)
+ + sizeof(rec->rid)
+ + sizeof(rec->qual)
+ + varint_size(rec->rlen)
+ + varint_size(2) // n_info
+ + varint_size(2) // n_allele
+ + varint_size(1) // n_fmt
+ + varint_size(3) // n_sample
+ + varint_size(rec->shared.l)
+ + varint_size(rec->indiv.l)
+ + rec->shared.l + rec->indiv.l
+ + rec->unpack_size[1] // Alleles
+ 8; // the number of _align_up() calls
if ( delta > args->max_mem - args->mem )
{
+ packed_bcf_t *tmp = malloc(sizeof(*tmp) + rec->unpack_size[1] * sizeof(bcf1_t *));
+ if (!tmp)
+ clean_files_and_throw(args, "[%s] Out of memory\n", __func__);
+ pack_bcf_data(tmp, rec, 1);
+
args->nbuf++;
hts_expand(bcf1_t*, args->nbuf, args->mbuf, args->buf);
- args->buf[args->nbuf-1] = rec;
- buf_flush(args);
+ args->buf[args->nbuf-1] = tmp;
+
+ buf_flush(args, rec);
+
+ free(tmp);
bcf_destroy(rec);
return;
}
uint8_t *ptr_beg = args->mem_block + args->mem;
uint8_t *ptr = _align_up(ptr_beg);
- bcf1_t *new_rec = (bcf1_t*)ptr;
- memcpy(new_rec,rec,sizeof(*rec));
- ptr += sizeof(*rec);
-
- // The array of allele pointers does not need alignment as bcf1_t is already padded to the biggest
- // data type in the structure
- char **allele = (char**)ptr;
- ptr += rec->n_allele*sizeof(*allele);
-
- // This is just to prevent valgrind from complaining about memcpy, unpack_size is a high-water mark
- // and the end may be uninitialized
- delta = rec->d.allele[rec->n_allele-1] - rec->d.allele[0];
- while ( delta < rec->unpack_size[1] ) if ( !rec->d.als[delta++] ) break;
- memcpy(ptr,rec->d.als,delta);
- new_rec->d.als = (char*)ptr;
- ptr = ptr + delta;
+ packed_bcf_t *packed_rec = (packed_bcf_t *) ptr;
- int i;
- for (i=0; i<rec->n_allele; i++) allele[i] = new_rec->d.als + (ptrdiff_t)(rec->d.allele[i] - rec->d.allele[0]);
- new_rec->d.allele = allele;
-
- memcpy(ptr,rec->shared.s,rec->shared.l);
- new_rec->shared.s = (char*)ptr;
- new_rec->shared.m = rec->shared.l;
- ptr += rec->shared.l;
-
- memcpy(ptr,rec->indiv.s,rec->indiv.l);
- new_rec->indiv.s = (char*)ptr;
- new_rec->indiv.m = rec->indiv.l;
- ptr += rec->indiv.l;
-
- // This is just to prevent valgrind from complaining about memcpy, unpack_size is a high-water mark
- // and the end may be uninitialized
- i = 0;
- while ( i < rec->unpack_size[0] ) if ( !rec->d.id[i++] ) break;
- memcpy(ptr,rec->d.id,i);
- new_rec->d.id = (char*)ptr;
- ptr += i;
+ ptr = pack_bcf_data(packed_rec, rec, 0);
args->nbuf++;
hts_expand(bcf1_t*, args->nbuf, args->mbuf, args->buf);
- args->buf[args->nbuf-1] = new_rec;
+ args->buf[args->nbuf-1] = packed_rec;
delta = ptr - ptr_beg;
args->mem += delta;
bcf_destroy(rec);
break;
}
- if ( rec->errcode ) clean_files_and_throw(args,"Error encountered while parsing the input at %s:%d\n",bcf_seqname(args->hdr,rec),rec->pos+1);
+ if ( rec->errcode ) clean_files_and_throw(args,"Error encountered while parsing the input at %s:%"PRIhts_pos"\n",bcf_seqname(args->hdr,rec),rec->pos+1);
bcf_unpack(rec, BCF_UN_STR);
buf_push(args, rec);
}
- buf_flush(args);
+ buf_flush(args, NULL);
free(args->buf);
if ( hts_close(in)!=0 ) clean_files_and_throw(args,"Close failed: %s\n", args->fname);
{
blk_t *a = *aptr;
blk_t *b = *bptr;
- int ret = cmp_bcf_pos(&a->rec, &b->rec);
+ int ret = cmp_bcf_pos_ref_alt(&a->rec, &b->rec);
if ( ret < 0 ) return 1;
+ if (ret == 0 && a->idx < b->idx) return 1;
return 0;
}
KHEAP_INIT(blk, blk_t*, blk_is_smaller)
void blk_read(args_t *args, khp_blk_t *bhp, bcf_hdr_t *hdr, blk_t *blk)
{
- if ( !blk->fh ) return;
- int ret = bcf_read(blk->fh, hdr, blk->rec);
+ int ret;
+ if (blk->is_merged)
+ {
+ if ( !blk->fh ) return;
+ ret = bcf_read(blk->fh, hdr, blk->rec);
+ }
+ else
+ {
+ if ( !blk->bgz ) return;
+ ret = read_packed_bcf(blk->bgz, blk->rec);
+ }
if ( ret < -1 ) clean_files_and_throw(args, "Error reading %s\n", blk->fname);
if ( ret == -1 )
{
- if ( hts_close(blk->fh)!=0 ) clean_files_and_throw(args, "Close failed: %s\n", blk->fname);
- blk->fh = 0;
- return;
+ if (blk->is_merged)
+ {
+ if ( hts_close(blk->fh)!=0 ) clean_files_and_throw(args, "Close failed: %s\n", blk->fname);
+ blk->fh = NULL;
+ return;
+ }
+ else
+ {
+ if ( bgzf_close(blk->bgz) != 0)
+ clean_files_and_throw(args, "Close failed: %s\n", blk->fname);
+ blk->bgz = NULL;
+ return;
+ }
}
bcf_unpack(blk->rec, BCF_UN_STR);
khp_insert(blk, bhp, &blk);
}
-void merge_blocks(args_t *args)
+void merge_blocks(args_t *args, htsFile *out, const char *output_fname,
+ int idx_fmt, size_t from)
{
- fprintf(stderr,"Merging %d temporary files\n", (int)args->nblk);
khp_blk_t *bhp = khp_init(blk);
+ char *index_fn = NULL;
+ size_t i;
- int i;
- for (i=0; i<args->nblk; i++)
+ for (i=from; i<args->nblk; i++)
{
- blk_t *blk = args->blk + i;
- blk->fh = hts_open(blk->fname, "r");
- if ( !blk->fh ) clean_files_and_throw(args, "Could not read %s: %s\n", blk->fname, strerror(errno));
- bcf_hdr_t *hdr = bcf_hdr_read(blk->fh);
- bcf_hdr_destroy(hdr);
- blk->rec = bcf_init();
+ blk_t *blk = &args->blk[i];
+ if (blk->is_merged)
+ {
+ blk->fh = hts_open(blk->fname, "r");
+ if ( !blk->fh ) clean_files_and_throw(args, "Could not read %s: %s\n", blk->fname, strerror(errno));
+ bcf_hdr_t *hdr = bcf_hdr_read(blk->fh);
+ bcf_hdr_destroy(hdr);
+ }
+ else
+ {
+ blk->bgz = bgzf_open(blk->fname, "r");
+ if (!blk->bgz)
+ clean_files_and_throw(args, "Could not read %s: %s\n", blk->fname, strerror(errno));
+ }
blk_read(args, bhp, args->hdr, blk);
}
- char wmode[8];
- set_wmode(wmode,args->output_type,args->output_fname,args->clevel);
- htsFile *out = hts_open(args->output_fname ? args->output_fname : "-", wmode);
- if ( bcf_hdr_write(out, args->hdr)!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(out,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( bcf_hdr_write(out, args->hdr)!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__, output_fname);
+
+ if (idx_fmt) {
+ if ( init_index2(out,args->hdr,output_fname,&index_fn,idx_fmt)<0 )
+ error("Error: failed to initialise index for %s\n",output_fname);
+ }
+
while ( bhp->ndat )
{
blk_t *blk = bhp->dat[0];
khp_delete(blk, bhp);
blk_read(args, bhp, args->hdr, blk);
}
- if ( args->write_index )
+ if ( idx_fmt )
{
if ( bcf_idx_save(out)<0 )
{
- if ( hts_close(out)!=0 ) error("Error: close failed .. %s\n", args->output_fname?args->output_fname:"stdout");
- error("Error: cannot write to index %s\n", args->index_fn);
+ if ( hts_close(out)!=0 ) error("Error: close failed .. %s\n", output_fname);
+ error("Error: cannot write to index %s\n", index_fn);
}
- free(args->index_fn);
+ free(index_fn);
}
- if ( hts_close(out)!=0 ) clean_files_and_throw(args, "Close failed: %s\n", args->output_fname);
- clean_files(args);
+ for (i = from; i < args->nblk; i++)
+ {
+ blk_t *blk = &args->blk[i];
+ if (unlink(blk->fname) != 0)
+ clean_files_and_throw(args, "Couldn't remove temporary file %s\n", blk->fname);
+ free(blk->fname);
+ blk->fname = NULL;
+ }
- free(args->blk);
khp_destroy(blk, bhp);
+}
+
+void do_partial_merge(args_t *args)
+{
+ uint32_t to_layer = 0;
+ size_t to_merge = 0;
+
+ // Temp. files are arranged in layers of at most MAX_TMP_FILES_PER_LAYER.
+ // When a layer is full, it is merged into the next layer up. Each
+ // layer will therefore contain files with exponentially more records
+ // then the previous one, but will be merged exponentially less frequently.
+ // The result is that the overall complexity will remain O(n*log(n))
+ // even if we need to do lots of partial merges.
+
+ while (to_layer < MERGE_LAYERS
+ && args->tmp_layers[to_layer] >= MAX_TMP_FILES_PER_LAYER)
+ {
+ to_merge += args->tmp_layers[to_layer];
+ args->tmp_layers[to_layer] = 0;
+ to_layer++;
+ }
+
+ assert(to_merge > 0 && to_merge <= args->nblk);
+
+ if (to_layer == MERGE_LAYERS) {
+ // Edge case - if we've got here, we've completely used the
+ // temp file allocation, so merge absolutely everything and
+ // leave one file at the highest level. Strictly this breaks
+ // the O(n*log(n)) complexity, but unless MERGE_LAYERS and
+ // MAX_TMP_FILES_PER_LAYER are too small it would take so long
+ // to get here it should never actually happen...
+ assert(to_merge == MAX_TMP_FILES_PER_LAYER * MERGE_LAYERS);
+ to_layer = MERGE_LAYERS - 1;
+ }
+
+ blk_t tmp = { NULL };
+ open_tmp_file(args, &tmp, 1);
+ merge_blocks(args, tmp.fh, tmp.fname, 0, args->nblk - to_merge);
+ if (hts_close(tmp.fh) != 0)
+ clean_files_and_throw(args, "Close failed: %s\n", tmp.fname);
+
+ args->nblk -= to_merge;
+ assert(args->blk[args->nblk].fh == NULL);
+ assert(args->blk[args->nblk].fname == NULL);
+ args->blk[args->nblk].is_merged = 1;
+ args->blk[args->nblk].idx = tmp.idx;
+ args->blk[args->nblk++].fname = tmp.fname;
+ args->tmp_layers[to_layer]++;
+}
+
+void merge_to_output(args_t *args)
+{
+ char wmode[8] = { 0 };
+ set_wmode(wmode,args->output_type,args->output_fname,args->clevel);
+ const char *output_fname = args->output_fname ? args->output_fname : "-";
+
+ htsFile *out = hts_open(output_fname, wmode);
+ if (!out) clean_files_and_throw(args, "[%s] Error: cannot open %s\n", __func__, output_fname);
+
+ fprintf(stderr,"Merging %zd temporary files\n", args->nblk);
+ merge_blocks(args, out, output_fname, args->write_index, 0);
fprintf(stderr,"Done\n");
+
+ if ( hts_close(out)!=0 )
+ clean_files_and_throw(args, "Close failed: %s\n", output_fname);
+
+ clean_files(args);
}
static void usage(args_t *args)
fprintf(stderr, "Options:\n");
fprintf(stderr, " -m, --max-mem FLOAT[kMG] maximum memory to use [768M]\n"); // using metric units, 1M=1e6
fprintf(stderr, " -o, --output FILE output file name [stdout]\n");
- fprintf(stderr, " -O, --output-type b|u|z|v b: compressed BCF, u: uncompressed BCF, z: compressed VCF, v: uncompressed VCF [v]\n");
fprintf(stderr, " -O, --output-type u|b|v|z[0-9] u/b: un/compressed BCF, v/z: un/compressed VCF, 0-9: compression level [v]\n");
#ifdef _WIN32
#else
fprintf(stderr, " -T, --temp-dir DIR temporary files [/tmp/bcftools.XXXXXX]\n");
#endif
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
exit(1);
}
return mem;
}
-void mkdir_p(const char *fmt, ...);
static void init(args_t *args)
{
+ size_t i;
args->max_mem *= 0.9;
args->mem_block = malloc(args->max_mem);
if ( !args->mem_block ) error("Error: could not allocate %zu bytes of memory, try reducing --max-mem\n",args->max_mem);
args->mem = 0;
+ for (i = 0; i < MAX_TMP_FILES; i++)
+ {
+ args->blk[i].fname = NULL;
+ args->blk[i].rec = bcf_init();
+ if (!args->blk[i].rec)
+ clean_files_and_throw(args,"Couldn't allocate bcf record\n");
+ }
+
+
args->tmp_dir = init_tmp_prefix(args->tmp_dir);
#ifdef _WIN32
{"output-file",required_argument,NULL,'o'},
{"output",required_argument,NULL,'o'},
{"help",no_argument,NULL,'h'},
- {"write-index",no_argument,NULL,1},
+ {"write-index",optional_argument,NULL,'W'},
{0,0,0,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "m:T:O:o:h?",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "m:T:O:o:W::h?",loptions,NULL)) >= 0)
{
switch (c)
{
if ( *tmp || args->clevel<0 || args->clevel>9 ) error("Could not parse argument: --compression-level %s\n", optarg+1);
}
break;
- case 1 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
init(args);
sort_blocks(args);
- merge_blocks(args);
+ merge_to_output(args);
destroy(args);
return 0;
/* vcfsort.c -- sort subcommand
- Copyright (C) 2017-2023 Genome Research Ltd.
+ Copyright (C) 2017-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
#include <htslib/vcf.h>
#include <htslib/kstring.h>
#include <htslib/hts_os.h>
+#include <htslib/hts_defs.h>
+#include <htslib/bgzf.h>
#include "kheap.h"
#include "bcftools.h"
+#define MAX_TMP_FILES_PER_LAYER 32
+#define MERGE_LAYERS 12
+#define MAX_TMP_FILES (MAX_TMP_FILES_PER_LAYER * MERGE_LAYERS)
+
typedef struct
{
char *fname;
htsFile *fh;
+ BGZF *bgz;
+ size_t idx;
bcf1_t *rec;
+ int is_merged;
}
blk_t;
+typedef struct
+{
+ size_t len;
+ hts_pos_t pos;
+ int rid;
+ float qual;
+ uint8_t data[];
+}
+packed_bcf_t;
+
typedef struct _args_t
{
bcf_hdr_t *hdr;
char **argv, *fname, *output_fname, *tmp_dir;
int argc, output_type, clevel;
size_t max_mem, mem;
- bcf1_t **buf;
+ packed_bcf_t **buf;
uint8_t *mem_block;
- size_t nbuf, mbuf, nblk;
- blk_t *blk;
- char *index_fn;
+
+ size_t nbuf, mbuf, nblk, tmp_count;
+ blk_t blk[MAX_TMP_FILES];
+ uint32_t tmp_layers[MERGE_LAYERS];
int write_index;
}
args_t;
{
int i;
fprintf(bcftools_stderr,"Cleaning\n");
- for (i=0; i<args->nblk; i++)
+ for (i=0; i<MAX_TMP_FILES; i++)
{
- blk_t *blk = args->blk + i;
+ blk_t *blk = &args->blk[i];
if ( blk->fname )
{
unlink(blk->fname);
}
rmdir(args->tmp_dir);
}
-void clean_files_and_throw(args_t *args, const char *format, ...)
+void HTS_FORMAT(HTS_PRINTF_FMT, 2, 3) HTS_NORETURN
+clean_files_and_throw(args_t *args, const char *format, ...)
{
va_list ap;
va_start(ap, format);
}
int cmp_bcf_pos(const void *aptr, const void *bptr)
+{
+ bcf1_t *a = *((bcf1_t**)aptr);
+ bcf1_t *b = *((bcf1_t**)bptr);
+ if ( a->rid < b->rid ) return -1;
+ if ( a->rid > b->rid ) return 1;
+ if ( a->pos < b->pos ) return -1;
+ if ( a->pos > b->pos ) return 1;
+ return 0;
+}
+int cmp_bcf_pos_ref_alt(const void *aptr, const void *bptr)
{
bcf1_t *a = *((bcf1_t**)aptr);
bcf1_t *b = *((bcf1_t**)bptr);
return 0;
}
-void buf_flush(args_t *args)
+static int cmp_packed_bcf_pos_ref_alt(const void *aptr, const void *bptr)
{
- if ( !args->nbuf ) return;
+ packed_bcf_t *a = *(packed_bcf_t **) aptr;
+ packed_bcf_t *b = *(packed_bcf_t **) bptr;
- qsort(args->buf, args->nbuf, sizeof(*args->buf), cmp_bcf_pos);
+ if ( a->rid < b->rid ) return -1;
+ if ( a->rid > b->rid ) return 1;
+ if ( a->pos < b->pos ) return -1;
+ if ( a->pos > b->pos ) return 1;
+
+ // Sort lexicographically by ref,alt. These are stored tab-separated
+ // as the first item in packed_bcf_t::data
+ return strcmp((char *) a->data, (char *) b->data);
+}
- args->nblk++;
- args->blk = (blk_t*) realloc(args->blk, sizeof(blk_t)*args->nblk);
- if ( !args->blk ) error("Error: could not allocate %zu bytes of memory, try reducing --max-mem\n",sizeof(blk_t)*args->nblk);
- blk_t *blk = args->blk + args->nblk - 1;
+static int cmp_packed_bcf_pos_ref_alt_stable(const void *aptr, const void *bptr)
+{
+ // cmp_bcf_pos_ref_alt() with tie-breaker to make qsort stable
+ int res = cmp_packed_bcf_pos_ref_alt(aptr, bptr);
+ if (res != 0) return res;
+
+ // Got a tie - use the position in the original input to break it.
+ // As everything is read into a big memory buffer, for most records
+ // we can just compare the pointers directly. The exception is
+ // any record that didn't quite fit in the memory buffer, causing it to be
+ // flushed. Those are flagged by setting packed_bcf_t::len = SIZE_MAX, and
+ // as they were the last record in the segment, they should always sort
+ // after unflagged records.
+
+ packed_bcf_t *a = *(packed_bcf_t **) aptr;
+ packed_bcf_t *b = *(packed_bcf_t **) bptr;
+
+ if (a->len == SIZE_MAX) return 1;
+ if (b->len == SIZE_MAX) return -1;
+
+ return a < b ? -1 : 1;
+}
+
+static uint8_t *pack_unsigned(uint8_t *data, uint64_t val)
+{
+ do {
+ *data++ = (val & 0x7f) | ((val > 0x7f) ? 0x80 : 0);
+ val >>= 7;
+ } while (val > 0);
+ return data;
+}
+
+static uint8_t *pack_hts_pos(uint8_t *data, hts_pos_t val)
+{
+ uint64_t sign = val < 0;
+ uint64_t v = val < 0 ? -(val + 1) : val;
+ v = v << 1 | sign;
+ return pack_unsigned(data, v);
+}
+
+static uint8_t *pack_bcf_data(packed_bcf_t *dest, const bcf1_t *src,
+ int outside_buffer)
+{
+ uint32_t i;
+ uint8_t *data = dest->data;
+ uint8_t *start = dest->data;
+ dest->pos = src->pos;
+ dest->rid = src->rid;
+ dest->qual = src->qual;
+
+ // Copy in alleles, for the comparison function
+ for (i = 0; i < src->n_allele; i++)
+ {
+ size_t l = strlen(src->d.allele[i]);
+ if (i > 0) *data++ = '\t';
+ memcpy(data, src->d.allele[i], l);
+ data += l;
+ }
+ *data++ = '\0';
+
+ if (outside_buffer)
+ {
+ dest->len = SIZE_MAX;
+ memcpy(data, &src, sizeof(src));
+ data += sizeof(src);
+ return data;
+ }
+
+ data = pack_hts_pos(data, src->rlen);
+ data = pack_unsigned(data, src->n_info);
+ data = pack_unsigned(data, src->n_allele);
+ data = pack_unsigned(data, src->n_fmt);
+ data = pack_unsigned(data, src->n_sample);
+ data = pack_unsigned(data, src->shared.l);
+ data = pack_unsigned(data, src->indiv.l);
+ if (src->shared.l)
+ memcpy(data, src->shared.s, src->shared.l);
+ data += src->shared.l;
+ if (src->indiv.l)
+ memcpy(data, src->indiv.s, src->indiv.l);
+ data += src->indiv.l;
+ dest->len = data - start;
+ return data;
+}
+
+static int write_packed_bcf(BGZF *fp, packed_bcf_t *src)
+{
+ // Write pos, rid, qual
+ size_t len = src->data - (uint8_t *) &src->pos;
+ if (bgzf_write_small(fp, &src->pos, len) < len)
+ return -1;
+
+ // Skip the copy of the alleles
+ size_t skip = strlen((char *) src->data) + 1;
+
+ // Write everything else
+ if (src->len < SIZE_MAX)
+ {
+ // In main memory block
+ len = src->len - skip;
+ if (bgzf_write_small(fp, src->data + skip, len) < len)
+ return -1;
+ }
+ else
+ {
+ // Record didn't fit in the main block. To minimize the
+ // overflow, its packed_bcf_t data will be imcomplete. A pointer to
+ // its bcf1_t struct will have been placed after the allele data
+ // so we can finish the packing job and write it in the same format
+ // the rest of the data
+ bcf1_t *rec;
+ uint8_t tmp[100], *data = tmp;
+ memcpy(&rec, src->data + skip, sizeof(rec));
+
+ data = pack_hts_pos(data, rec->rlen);
+ data = pack_unsigned(data, rec->n_info);
+ data = pack_unsigned(data, rec->n_allele);
+ data = pack_unsigned(data, rec->n_fmt);
+ data = pack_unsigned(data, rec->n_sample);
+ data = pack_unsigned(data, rec->shared.l);
+ data = pack_unsigned(data, rec->indiv.l);
+ if (bgzf_write_small(fp, tmp, data - tmp) < data - tmp)
+ return -1;
+ if (rec->shared.l > 0 &&
+ bgzf_write_small(fp, rec->shared.s, rec->shared.l) < rec->shared.l)
+ return -1;
+ if (rec->indiv.l > 0 &&
+ bgzf_write_small(fp, rec->indiv.s, rec->indiv.l) < rec->indiv.l)
+ return -1;
+ }
+
+ return 0;
+}
+
+static uint64_t unpack_unsigned(BGZF *fp, int *err)
+{
+ uint8_t data;
+ uint64_t val = 0;
+ uint32_t i = 0;
+
+ if (bgzf_read_small(fp, &data, sizeof(data)) <= 0)
+ goto short_read;
+
+ while (data & 0x80)
+ {
+ val |= (uint64_t)(data & 0x7f) << i;
+ i += 7;
+ if (bgzf_read_small(fp, &data, sizeof(data)) <= 0)
+ goto short_read;
+ }
+ val |= (uint64_t)data << i;
+ return val;
+
+ short_read:
+ *err = 1;
+ return 0;
+}
+
+static hts_pos_t unpack_hts_pos(BGZF *fp, int *err)
+{
+ uint64_t v = unpack_unsigned(fp, err);
+
+ if ((v & 1) == 0)
+ return (hts_pos_t)(v >> 1);
+ else
+ return -(hts_pos_t)(v >> 1) - 1;
+}
+
+static int read_packed_bcf(BGZF *fp, bcf1_t *dest)
+{
+ int err = 0;
+ packed_bcf_t tmp;
+ size_t len = tmp.data - (uint8_t *) &tmp.pos;
+
+ bcf_clear(dest);
+ ssize_t got = bgzf_read_small(fp, &tmp.pos, len);
+ if (got == 0)
+ return -1; // EOF
+ if (got < len)
+ return -2; // Error or short read
+ dest->pos = tmp.pos;
+ dest->rid = tmp.rid;
+ dest->qual = tmp.qual;
+ dest->rlen = unpack_hts_pos(fp, &err);
+ dest->n_info = unpack_unsigned(fp, &err);
+ dest->n_allele = unpack_unsigned(fp, &err);
+ dest->n_fmt = unpack_unsigned(fp, &err);
+ dest->n_sample = unpack_unsigned(fp, &err);
+ len = unpack_unsigned(fp, &err);
+ if (ks_resize(&dest->shared, len) != 0)
+ return -2;
+ dest->shared.l = len;
+ len = unpack_unsigned(fp, &err);
+ if (ks_resize(&dest->indiv, len) != 0)
+ return -2;
+ dest->indiv.l = len;
+ err |= bgzf_read_small(fp, dest->shared.s, dest->shared.l) < dest->shared.l;
+ err |= bgzf_read_small(fp, dest->indiv.s, dest->indiv.l) < dest->indiv.l;
+ return err == 0 ? 0 : -2;
+}
+void open_tmp_file(args_t *args, blk_t *blk, int is_merged)
+{
kstring_t str = {0,0,0};
- ksprintf(&str, "%s/%05d.bcf", args->tmp_dir, (int)args->nblk);
- blk->fname = str.s;
- blk->rec = NULL;
- blk->fh = NULL;
+ int tries = 1000;
- htsFile *fh = hts_open(blk->fname, "wbu");
- if ( fh == NULL ) clean_files_and_throw(args, "Cannot write %s: %s\n", blk->fname, strerror(errno));
- if ( bcf_hdr_write(fh, args->hdr)!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,blk->fname);
+ blk->fh = NULL;
+ blk->bgz = NULL;
+
+ do {
+ if (ksprintf(ks_clear(&str), "%s/%05zd%s",
+ args->tmp_dir, args->tmp_count++,
+ is_merged ? ".bcf" : "") < 0) {
+ clean_files_and_throw(args, "%s", strerror(errno));
+ }
+
+ if (is_merged)
+ blk->fh = hts_open(str.s, "wbx1");
+ else
+ blk->bgz = bgzf_open(str.s, "wx1");
+ if ( blk->fh == NULL && blk->bgz == NULL && (errno != EEXIST || --tries <= 0)) {
+ clean_files_and_throw(args, "Cannot write %s: %s\n",
+ str.s, strerror(errno));
+ }
+ } while (blk->fh == NULL && blk->bgz == NULL);
+
+ blk->fname = ks_release(&str);
+ blk->idx = args->tmp_count - 1;
+}
+
+void do_partial_merge(args_t *args);
+
+void buf_flush(args_t *args, bcf1_t *last_rec)
+{
+ if ( !args->nbuf ) return;
+
+ qsort(args->buf, args->nbuf, sizeof(*args->buf), cmp_packed_bcf_pos_ref_alt_stable);
+
+ if (args->tmp_layers[0] >= MAX_TMP_FILES_PER_LAYER)
+ do_partial_merge(args);
+
+ assert(args->nblk < MAX_TMP_FILES);
+ blk_t *blk = &args->blk[args->nblk];
+ blk->is_merged = 0;
+ args->nblk++;
+ args->tmp_layers[0]++;
+ assert(blk->fname == NULL && blk->fh == NULL && blk->bgz == NULL);
+
+ open_tmp_file(args, blk, 0);
int i;
for (i=0; i<args->nbuf; i++)
{
- if ( bcf_write(fh, args->hdr, args->buf[i])!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,blk->fname);
+ if ( write_packed_bcf(blk->bgz, args->buf[i])!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,blk->fname);
}
- if ( hts_close(fh)!=0 ) clean_files_and_throw(args, "[%s] Error: close failed .. %s\n", __func__,blk->fname);
+
+ if ( bgzf_close(blk->bgz)!=0 ) clean_files_and_throw(args, "[%s] Error: close failed .. %s\n", __func__,blk->fname);
+ blk->bgz = NULL;
args->nbuf = 0;
args->mem = 0;
return (uint8_t*)(((size_t)ptr + 8 - 1) & ~((size_t)(8 - 1)));
}
+#define varint_size(X) ((sizeof(X) * 8 + 7) / 7) // worst case
+
void buf_push(args_t *args, bcf1_t *rec)
{
- size_t delta = sizeof(bcf1_t) + rec->shared.l + rec->indiv.l + rec->unpack_size[0] + rec->unpack_size[1]
- + sizeof(*rec->d.allele)*rec->d.m_allele
- + sizeof(bcf1_t*) // args->buf
+ size_t delta = sizeof(rec->pos)
+ + sizeof(rec->rid)
+ + sizeof(rec->qual)
+ + varint_size(rec->rlen)
+ + varint_size(2) // n_info
+ + varint_size(2) // n_allele
+ + varint_size(1) // n_fmt
+ + varint_size(3) // n_sample
+ + varint_size(rec->shared.l)
+ + varint_size(rec->indiv.l)
+ + rec->shared.l + rec->indiv.l
+ + rec->unpack_size[1] // Alleles
+ 8; // the number of _align_up() calls
if ( delta > args->max_mem - args->mem )
{
+ packed_bcf_t *tmp = malloc(sizeof(*tmp) + rec->unpack_size[1] * sizeof(bcf1_t *));
+ if (!tmp)
+ clean_files_and_throw(args, "[%s] Out of memory\n", __func__);
+ pack_bcf_data(tmp, rec, 1);
+
args->nbuf++;
hts_expand(bcf1_t*, args->nbuf, args->mbuf, args->buf);
- args->buf[args->nbuf-1] = rec;
- buf_flush(args);
+ args->buf[args->nbuf-1] = tmp;
+
+ buf_flush(args, rec);
+
+ free(tmp);
bcf_destroy(rec);
return;
}
uint8_t *ptr_beg = args->mem_block + args->mem;
uint8_t *ptr = _align_up(ptr_beg);
- bcf1_t *new_rec = (bcf1_t*)ptr;
- memcpy(new_rec,rec,sizeof(*rec));
- ptr += sizeof(*rec);
-
- // The array of allele pointers does not need alignment as bcf1_t is already padded to the biggest
- // data type in the structure
- char **allele = (char**)ptr;
- ptr += rec->n_allele*sizeof(*allele);
-
- // This is just to prevent valgrind from complaining about memcpy, unpack_size is a high-water mark
- // and the end may be uninitialized
- delta = rec->d.allele[rec->n_allele-1] - rec->d.allele[0];
- while ( delta < rec->unpack_size[1] ) if ( !rec->d.als[delta++] ) break;
- memcpy(ptr,rec->d.als,delta);
- new_rec->d.als = (char*)ptr;
- ptr = ptr + delta;
+ packed_bcf_t *packed_rec = (packed_bcf_t *) ptr;
- int i;
- for (i=0; i<rec->n_allele; i++) allele[i] = new_rec->d.als + (ptrdiff_t)(rec->d.allele[i] - rec->d.allele[0]);
- new_rec->d.allele = allele;
-
- memcpy(ptr,rec->shared.s,rec->shared.l);
- new_rec->shared.s = (char*)ptr;
- new_rec->shared.m = rec->shared.l;
- ptr += rec->shared.l;
-
- memcpy(ptr,rec->indiv.s,rec->indiv.l);
- new_rec->indiv.s = (char*)ptr;
- new_rec->indiv.m = rec->indiv.l;
- ptr += rec->indiv.l;
-
- // This is just to prevent valgrind from complaining about memcpy, unpack_size is a high-water mark
- // and the end may be uninitialized
- i = 0;
- while ( i < rec->unpack_size[0] ) if ( !rec->d.id[i++] ) break;
- memcpy(ptr,rec->d.id,i);
- new_rec->d.id = (char*)ptr;
- ptr += i;
+ ptr = pack_bcf_data(packed_rec, rec, 0);
args->nbuf++;
hts_expand(bcf1_t*, args->nbuf, args->mbuf, args->buf);
- args->buf[args->nbuf-1] = new_rec;
+ args->buf[args->nbuf-1] = packed_rec;
delta = ptr - ptr_beg;
args->mem += delta;
bcf_destroy(rec);
break;
}
- if ( rec->errcode ) clean_files_and_throw(args,"Error encountered while parsing the input at %s:%d\n",bcf_seqname(args->hdr,rec),rec->pos+1);
+ if ( rec->errcode ) clean_files_and_throw(args,"Error encountered while parsing the input at %s:%"PRIhts_pos"\n",bcf_seqname(args->hdr,rec),rec->pos+1);
bcf_unpack(rec, BCF_UN_STR);
buf_push(args, rec);
}
- buf_flush(args);
+ buf_flush(args, NULL);
free(args->buf);
if ( hts_close(in)!=0 ) clean_files_and_throw(args,"Close failed: %s\n", args->fname);
{
blk_t *a = *aptr;
blk_t *b = *bptr;
- int ret = cmp_bcf_pos(&a->rec, &b->rec);
+ int ret = cmp_bcf_pos_ref_alt(&a->rec, &b->rec);
if ( ret < 0 ) return 1;
+ if (ret == 0 && a->idx < b->idx) return 1;
return 0;
}
KHEAP_INIT(blk, blk_t*, blk_is_smaller)
void blk_read(args_t *args, khp_blk_t *bhp, bcf_hdr_t *hdr, blk_t *blk)
{
- if ( !blk->fh ) return;
- int ret = bcf_read(blk->fh, hdr, blk->rec);
+ int ret;
+ if (blk->is_merged)
+ {
+ if ( !blk->fh ) return;
+ ret = bcf_read(blk->fh, hdr, blk->rec);
+ }
+ else
+ {
+ if ( !blk->bgz ) return;
+ ret = read_packed_bcf(blk->bgz, blk->rec);
+ }
if ( ret < -1 ) clean_files_and_throw(args, "Error reading %s\n", blk->fname);
if ( ret == -1 )
{
- if ( hts_close(blk->fh)!=0 ) clean_files_and_throw(args, "Close failed: %s\n", blk->fname);
- blk->fh = 0;
- return;
+ if (blk->is_merged)
+ {
+ if ( hts_close(blk->fh)!=0 ) clean_files_and_throw(args, "Close failed: %s\n", blk->fname);
+ blk->fh = NULL;
+ return;
+ }
+ else
+ {
+ if ( bgzf_close(blk->bgz) != 0)
+ clean_files_and_throw(args, "Close failed: %s\n", blk->fname);
+ blk->bgz = NULL;
+ return;
+ }
}
bcf_unpack(blk->rec, BCF_UN_STR);
khp_insert(blk, bhp, &blk);
}
-void merge_blocks(args_t *args)
+void merge_blocks(args_t *args, htsFile *out, const char *output_fname,
+ int idx_fmt, size_t from)
{
- fprintf(bcftools_stderr,"Merging %d temporary files\n", (int)args->nblk);
khp_blk_t *bhp = khp_init(blk);
+ char *index_fn = NULL;
+ size_t i;
- int i;
- for (i=0; i<args->nblk; i++)
+ for (i=from; i<args->nblk; i++)
{
- blk_t *blk = args->blk + i;
- blk->fh = hts_open(blk->fname, "r");
- if ( !blk->fh ) clean_files_and_throw(args, "Could not read %s: %s\n", blk->fname, strerror(errno));
- bcf_hdr_t *hdr = bcf_hdr_read(blk->fh);
- bcf_hdr_destroy(hdr);
- blk->rec = bcf_init();
+ blk_t *blk = &args->blk[i];
+ if (blk->is_merged)
+ {
+ blk->fh = hts_open(blk->fname, "r");
+ if ( !blk->fh ) clean_files_and_throw(args, "Could not read %s: %s\n", blk->fname, strerror(errno));
+ bcf_hdr_t *hdr = bcf_hdr_read(blk->fh);
+ bcf_hdr_destroy(hdr);
+ }
+ else
+ {
+ blk->bgz = bgzf_open(blk->fname, "r");
+ if (!blk->bgz)
+ clean_files_and_throw(args, "Could not read %s: %s\n", blk->fname, strerror(errno));
+ }
blk_read(args, bhp, args->hdr, blk);
}
- char wmode[8];
- set_wmode(wmode,args->output_type,args->output_fname,args->clevel);
- htsFile *out = hts_open(args->output_fname ? args->output_fname : "-", wmode);
- if ( bcf_hdr_write(out, args->hdr)!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__,args->output_fname);
- if ( args->write_index && init_index(out,args->hdr,args->output_fname,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->output_fname);
+ if ( bcf_hdr_write(out, args->hdr)!=0 ) clean_files_and_throw(args, "[%s] Error: cannot write to %s\n", __func__, output_fname);
+
+ if (idx_fmt) {
+ if ( init_index2(out,args->hdr,output_fname,&index_fn,idx_fmt)<0 )
+ error("Error: failed to initialise index for %s\n",output_fname);
+ }
+
while ( bhp->ndat )
{
blk_t *blk = bhp->dat[0];
khp_delete(blk, bhp);
blk_read(args, bhp, args->hdr, blk);
}
- if ( args->write_index )
+ if ( idx_fmt )
{
if ( bcf_idx_save(out)<0 )
{
- if ( hts_close(out)!=0 ) error("Error: close failed .. %s\n", args->output_fname?args->output_fname:"bcftools_stdout");
- error("Error: cannot write to index %s\n", args->index_fn);
+ if ( hts_close(out)!=0 ) error("Error: close failed .. %s\n", output_fname);
+ error("Error: cannot write to index %s\n", index_fn);
}
- free(args->index_fn);
+ free(index_fn);
}
- if ( hts_close(out)!=0 ) clean_files_and_throw(args, "Close failed: %s\n", args->output_fname);
- clean_files(args);
+ for (i = from; i < args->nblk; i++)
+ {
+ blk_t *blk = &args->blk[i];
+ if (unlink(blk->fname) != 0)
+ clean_files_and_throw(args, "Couldn't remove temporary file %s\n", blk->fname);
+ free(blk->fname);
+ blk->fname = NULL;
+ }
- free(args->blk);
khp_destroy(blk, bhp);
+}
+
+void do_partial_merge(args_t *args)
+{
+ uint32_t to_layer = 0;
+ size_t to_merge = 0;
+
+ // Temp. files are arranged in layers of at most MAX_TMP_FILES_PER_LAYER.
+ // When a layer is full, it is merged into the next layer up. Each
+ // layer will therefore contain files with exponentially more records
+ // then the previous one, but will be merged exponentially less frequently.
+ // The result is that the overall complexity will remain O(n*log(n))
+ // even if we need to do lots of partial merges.
+
+ while (to_layer < MERGE_LAYERS
+ && args->tmp_layers[to_layer] >= MAX_TMP_FILES_PER_LAYER)
+ {
+ to_merge += args->tmp_layers[to_layer];
+ args->tmp_layers[to_layer] = 0;
+ to_layer++;
+ }
+
+ assert(to_merge > 0 && to_merge <= args->nblk);
+
+ if (to_layer == MERGE_LAYERS) {
+ // Edge case - if we've got here, we've completely used the
+ // temp file allocation, so merge absolutely everything and
+ // leave one file at the highest level. Strictly this breaks
+ // the O(n*log(n)) complexity, but unless MERGE_LAYERS and
+ // MAX_TMP_FILES_PER_LAYER are too small it would take so long
+ // to get here it should never actually happen...
+ assert(to_merge == MAX_TMP_FILES_PER_LAYER * MERGE_LAYERS);
+ to_layer = MERGE_LAYERS - 1;
+ }
+
+ blk_t tmp = { NULL };
+ open_tmp_file(args, &tmp, 1);
+ merge_blocks(args, tmp.fh, tmp.fname, 0, args->nblk - to_merge);
+ if (hts_close(tmp.fh) != 0)
+ clean_files_and_throw(args, "Close failed: %s\n", tmp.fname);
+
+ args->nblk -= to_merge;
+ assert(args->blk[args->nblk].fh == NULL);
+ assert(args->blk[args->nblk].fname == NULL);
+ args->blk[args->nblk].is_merged = 1;
+ args->blk[args->nblk].idx = tmp.idx;
+ args->blk[args->nblk++].fname = tmp.fname;
+ args->tmp_layers[to_layer]++;
+}
+
+void merge_to_output(args_t *args)
+{
+ char wmode[8] = { 0 };
+ set_wmode(wmode,args->output_type,args->output_fname,args->clevel);
+ const char *output_fname = args->output_fname ? args->output_fname : "-";
+
+ htsFile *out = hts_open(output_fname, wmode);
+ if (!out) clean_files_and_throw(args, "[%s] Error: cannot open %s\n", __func__, output_fname);
+
+ fprintf(bcftools_stderr,"Merging %zd temporary files\n", args->nblk);
+ merge_blocks(args, out, output_fname, args->write_index, 0);
fprintf(bcftools_stderr,"Done\n");
+
+ if ( hts_close(out)!=0 )
+ clean_files_and_throw(args, "Close failed: %s\n", output_fname);
+
+ clean_files(args);
}
static void usage(args_t *args)
fprintf(bcftools_stderr, "Options:\n");
fprintf(bcftools_stderr, " -m, --max-mem FLOAT[kMG] maximum memory to use [768M]\n"); // using metric units, 1M=1e6
fprintf(bcftools_stderr, " -o, --output FILE output file name [bcftools_stdout]\n");
- fprintf(bcftools_stderr, " -O, --output-type b|u|z|v b: compressed BCF, u: uncompressed BCF, z: compressed VCF, v: uncompressed VCF [v]\n");
fprintf(bcftools_stderr, " -O, --output-type u|b|v|z[0-9] u/b: un/compressed BCF, v/z: un/compressed VCF, 0-9: compression level [v]\n");
#ifdef _WIN32
#else
fprintf(bcftools_stderr, " -T, --temp-dir DIR temporary files [/tmp/bcftools.XXXXXX]\n");
#endif
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
return mem;
}
-void mkdir_p(const char *fmt, ...);
static void init(args_t *args)
{
+ size_t i;
args->max_mem *= 0.9;
args->mem_block = malloc(args->max_mem);
if ( !args->mem_block ) error("Error: could not allocate %zu bytes of memory, try reducing --max-mem\n",args->max_mem);
args->mem = 0;
+ for (i = 0; i < MAX_TMP_FILES; i++)
+ {
+ args->blk[i].fname = NULL;
+ args->blk[i].rec = bcf_init();
+ if (!args->blk[i].rec)
+ clean_files_and_throw(args,"Couldn't allocate bcf record\n");
+ }
+
+
args->tmp_dir = init_tmp_prefix(args->tmp_dir);
#ifdef _WIN32
{"output-file",required_argument,NULL,'o'},
{"output",required_argument,NULL,'o'},
{"help",no_argument,NULL,'h'},
- {"write-index",no_argument,NULL,1},
+ {"write-index",optional_argument,NULL,'W'},
{0,0,0,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "m:T:O:o:h?",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "m:T:O:o:W::h?",loptions,NULL)) >= 0)
{
switch (c)
{
if ( *tmp || args->clevel<0 || args->clevel>9 ) error("Could not parse argument: --compression-level %s\n", optarg+1);
}
break;
- case 1 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case 'h':
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
init(args);
sort_blocks(args);
- merge_blocks(args);
+ merge_to_output(args);
destroy(args);
return 0;
/* vcfstats.c -- Produces stats which can be plotted using plot-vcfstats.
- Copyright (C) 2012-2023 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
{
stats->nusr = args->nusr;
stats->usr = (user_stats_t*)malloc(sizeof(user_stats_t)*args->nusr);
- memcpy(stats->usr,args->usr,args->nusr*sizeof(user_stats_t));
+ if (args->nusr)
+ memcpy(stats->usr,args->usr,args->nusr*sizeof(user_stats_t));
int i;
for (i=0; i<stats->nusr; i++)
{
*ial = 0;
#define BRANCH_INT(type_t,missing,vector_end) { \
type_t *ptr = (type_t *) (ad_fmt_ptr->p + ad_fmt_ptr->size*ismpl); \
- for (iv=1; iv<ad_fmt_ptr->n; iv++) \
+ for (iv=1; iv<ad_fmt_ptr->n && iv<line->n_allele; iv++) \
{ \
if ( ptr[iv]==vector_end ) break; \
if ( ptr[iv]==missing ) continue; \
#define vaf2bin(vaf) ((int)nearbyintf((vaf)/0.05))
static inline void update_vaf(vaf_t *smpl_vaf, bcf1_t *line, int ial, float vaf)
{
- int idx = vaf2bin(vaf);
- if ( bcf_get_variant_type(line,ial)==VCF_SNP ) smpl_vaf->snv[idx]++;
- else smpl_vaf->indel[idx]++;
+ if ( vaf>=0 && vaf<=1 )
+ {
+ int idx = vaf2bin(vaf);
+ if ( bcf_get_variant_type(line,ial)==VCF_SNP ) smpl_vaf->snv[idx]++;
+ else smpl_vaf->indel[idx]++;
+ }
}
static inline int calc_sample_depth(args_t *args, int ismpl, bcf_fmt_t *ad_fmt_ptr, bcf_fmt_t *dp_fmt_ptr)
printf("SN\t%d\tnumber of multiallelic sites:\t%"PRIu64"\n", id, stats->n_mals);
printf("SN\t%d\tnumber of multiallelic SNP sites:\t%"PRIu64"\n", id, stats->n_snp_mals);
}
- printf("# TSTV, transitions/transversions:\n# TSTV\t[2]id\t[3]ts\t[4]tv\t[5]ts/tv\t[6]ts (1st ALT)\t[7]tv (1st ALT)\t[8]ts/tv (1st ALT)\n");
+ printf("# TSTV, transitions/transversions\n"
+ "# - transitions, see https://en.wikipedia.org/wiki/Transition_(genetics)\n"
+ "# - transversions, see https://en.wikipedia.org/wiki/Transversion\n");
+ printf("# TSTV\t[2]id\t[3]ts\t[4]tv\t[5]ts/tv\t[6]ts (1st ALT)\t[7]tv (1st ALT)\t[8]ts/tv (1st ALT)\n");
for (id=0; id<args->nstats; id++)
{
stats_t *stats = &args->stats[id];
}
if ( args->indel_ctx )
{
- printf("# ICS, Indel context summary:\n# ICS\t[2]id\t[3]repeat-consistent\t[4]repeat-inconsistent\t[5]not applicable\t[6]c/(c+i) ratio\n");
+ printf("# ICS, Indel context:\n"
+ "# - repeat-consistent, inconsistent and n/a: experimental and useless stats [DEPRECATED]\n");
+ printf("# ICS\t[2]id\t[3]repeat-consistent\t[4]repeat-inconsistent\t[5]not applicable\t[6]c/(c+i) ratio\n");
for (id=0; id<args->nstats; id++)
{
int nc = 0, ni = 0, na = args->stats[id].n_repeat_na;
}
printf("ICS\t%d\t%d\t%d\t%d\t%.4f\n", id, nc,ni,na,nc+ni ? (float)nc/(nc+ni) : 0.0);
}
- printf("# ICL, Indel context by length:\n# ICL\t[2]id\t[3]length of repeat element\t[4]repeat-consistent deletions)\t[5]repeat-inconsistent deletions\t[6]consistent insertions\t[7]inconsistent insertions\t[8]c/(c+i) ratio\n");
+ printf("# ICL, Indel context by length:\n"
+ "# - repeat-consistent, inconsistent and n/a: experimental and useless stats [DEPRECATED]\n");
+ printf("# ICL\t[2]id\t[3]length of repeat element\t[4]repeat-consistent deletions)\t[5]repeat-inconsistent deletions\t[6]consistent insertions\t[7]inconsistent insertions\t[8]c/(c+i) ratio\n");
for (id=0; id<args->nstats; id++)
{
for (i=1; i<IRC_RLEN; i++)
}
}
}
- printf("# SiS, Singleton stats:\n# SiS\t[2]id\t[3]allele count\t[4]number of SNPs\t[5]number of transitions\t[6]number of transversions\t[7]number of indels\t[8]repeat-consistent\t[9]repeat-inconsistent\t[10]not applicable\n");
+ printf("# SiS, Singleton stats:\n"
+ "# - allele count, i.e. the number of singleton genotypes (AC=1)\n"
+ "# - number of transitions, see above\n"
+ "# - number of transversions, see above\n"
+ "# - repeat-consistent, inconsistent and n/a: experimental and useless stats [DEPRECATED]\n");
+ printf("# SiS\t[2]id\t[3]allele count\t[4]number of SNPs\t[5]number of transitions\t[6]number of transversions\t[7]number of indels\t[8]repeat-consistent\t[9]repeat-inconsistent\t[10]not applicable\n");
for (id=0; id<args->nstats; id++)
{
stats_t *stats = &args->stats[id];
{
if ( usr->vals_ts[j]+usr->vals_tv[j] == 0 ) continue; // skip empty bins
float val = usr->min + (usr->max - usr->min)*j/(usr->nbins-1);
- const char *fmt = usr->type==BCF_HT_REAL ? "USR:%s/%d\t%d\t%e\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n" : "USR:%s/%d\t%d\t%.0f\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n";
+ const char * const fmt = usr->type==BCF_HT_REAL ? "USR:%s/%d\t%d\t%e\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n" : "USR:%s/%d\t%d\t%.0f\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n";
printf(fmt,usr->tag,usr->idx,id,val,usr->vals_ts[j]+usr->vals_tv[j],usr->vals_ts[j],usr->vals_tv[j]);
}
}
}
}
+ printf("# DP, depth:\n"
+ "# - set id, see above\n"
+ "# - the depth bin, corresponds to the depth (unless --depth was given)\n"
+ "# - number of genotypes with this depth (zero unless -s/-S was given)\n"
+ "# - fraction of genotypes with this depth (zero unless -s/-S was given)\n"
+ "# - number of sites with this depth\n"
+ "# - fraction of sites with this depth\n");
printf("# DP, Depth distribution\n# DP\t[2]id\t[3]bin\t[4]number of genotypes\t[5]fraction of genotypes (%%)\t[6]number of sites\t[7]fraction of sites (%%)\n");
for (id=0; id<args->nstats; id++)
{
/* vcfstats.c -- Produces stats which can be plotted using plot-vcfstats.
- Copyright (C) 2012-2023 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
{
stats->nusr = args->nusr;
stats->usr = (user_stats_t*)malloc(sizeof(user_stats_t)*args->nusr);
- memcpy(stats->usr,args->usr,args->nusr*sizeof(user_stats_t));
+ if (args->nusr)
+ memcpy(stats->usr,args->usr,args->nusr*sizeof(user_stats_t));
int i;
for (i=0; i<stats->nusr; i++)
{
*ial = 0;
#define BRANCH_INT(type_t,missing,vector_end) { \
type_t *ptr = (type_t *) (ad_fmt_ptr->p + ad_fmt_ptr->size*ismpl); \
- for (iv=1; iv<ad_fmt_ptr->n; iv++) \
+ for (iv=1; iv<ad_fmt_ptr->n && iv<line->n_allele; iv++) \
{ \
if ( ptr[iv]==vector_end ) break; \
if ( ptr[iv]==missing ) continue; \
#define vaf2bin(vaf) ((int)nearbyintf((vaf)/0.05))
static inline void update_vaf(vaf_t *smpl_vaf, bcf1_t *line, int ial, float vaf)
{
- int idx = vaf2bin(vaf);
- if ( bcf_get_variant_type(line,ial)==VCF_SNP ) smpl_vaf->snv[idx]++;
- else smpl_vaf->indel[idx]++;
+ if ( vaf>=0 && vaf<=1 )
+ {
+ int idx = vaf2bin(vaf);
+ if ( bcf_get_variant_type(line,ial)==VCF_SNP ) smpl_vaf->snv[idx]++;
+ else smpl_vaf->indel[idx]++;
+ }
}
static inline int calc_sample_depth(args_t *args, int ismpl, bcf_fmt_t *ad_fmt_ptr, bcf_fmt_t *dp_fmt_ptr)
fprintf(bcftools_stdout, "SN\t%d\tnumber of multiallelic sites:\t%"PRIu64"\n", id, stats->n_mals);
fprintf(bcftools_stdout, "SN\t%d\tnumber of multiallelic SNP sites:\t%"PRIu64"\n", id, stats->n_snp_mals);
}
- fprintf(bcftools_stdout, "# TSTV, transitions/transversions:\n# TSTV\t[2]id\t[3]ts\t[4]tv\t[5]ts/tv\t[6]ts (1st ALT)\t[7]tv (1st ALT)\t[8]ts/tv (1st ALT)\n");
+ fprintf(bcftools_stdout, "# TSTV, transitions/transversions\n"
+ "# - transitions, see https://en.wikipedia.org/wiki/Transition_(genetics)\n"
+ "# - transversions, see https://en.wikipedia.org/wiki/Transversion\n");
+ fprintf(bcftools_stdout, "# TSTV\t[2]id\t[3]ts\t[4]tv\t[5]ts/tv\t[6]ts (1st ALT)\t[7]tv (1st ALT)\t[8]ts/tv (1st ALT)\n");
for (id=0; id<args->nstats; id++)
{
stats_t *stats = &args->stats[id];
}
if ( args->indel_ctx )
{
- fprintf(bcftools_stdout, "# ICS, Indel context summary:\n# ICS\t[2]id\t[3]repeat-consistent\t[4]repeat-inconsistent\t[5]not applicable\t[6]c/(c+i) ratio\n");
+ fprintf(bcftools_stdout, "# ICS, Indel context:\n"
+ "# - repeat-consistent, inconsistent and n/a: experimental and useless stats [DEPRECATED]\n");
+ fprintf(bcftools_stdout, "# ICS\t[2]id\t[3]repeat-consistent\t[4]repeat-inconsistent\t[5]not applicable\t[6]c/(c+i) ratio\n");
for (id=0; id<args->nstats; id++)
{
int nc = 0, ni = 0, na = args->stats[id].n_repeat_na;
}
fprintf(bcftools_stdout, "ICS\t%d\t%d\t%d\t%d\t%.4f\n", id, nc,ni,na,nc+ni ? (float)nc/(nc+ni) : 0.0);
}
- fprintf(bcftools_stdout, "# ICL, Indel context by length:\n# ICL\t[2]id\t[3]length of repeat element\t[4]repeat-consistent deletions)\t[5]repeat-inconsistent deletions\t[6]consistent insertions\t[7]inconsistent insertions\t[8]c/(c+i) ratio\n");
+ fprintf(bcftools_stdout, "# ICL, Indel context by length:\n"
+ "# - repeat-consistent, inconsistent and n/a: experimental and useless stats [DEPRECATED]\n");
+ fprintf(bcftools_stdout, "# ICL\t[2]id\t[3]length of repeat element\t[4]repeat-consistent deletions)\t[5]repeat-inconsistent deletions\t[6]consistent insertions\t[7]inconsistent insertions\t[8]c/(c+i) ratio\n");
for (id=0; id<args->nstats; id++)
{
for (i=1; i<IRC_RLEN; i++)
}
}
}
- fprintf(bcftools_stdout, "# SiS, Singleton stats:\n# SiS\t[2]id\t[3]allele count\t[4]number of SNPs\t[5]number of transitions\t[6]number of transversions\t[7]number of indels\t[8]repeat-consistent\t[9]repeat-inconsistent\t[10]not applicable\n");
+ fprintf(bcftools_stdout, "# SiS, Singleton stats:\n"
+ "# - allele count, i.e. the number of singleton genotypes (AC=1)\n"
+ "# - number of transitions, see above\n"
+ "# - number of transversions, see above\n"
+ "# - repeat-consistent, inconsistent and n/a: experimental and useless stats [DEPRECATED]\n");
+ fprintf(bcftools_stdout, "# SiS\t[2]id\t[3]allele count\t[4]number of SNPs\t[5]number of transitions\t[6]number of transversions\t[7]number of indels\t[8]repeat-consistent\t[9]repeat-inconsistent\t[10]not applicable\n");
for (id=0; id<args->nstats; id++)
{
stats_t *stats = &args->stats[id];
{
if ( usr->vals_ts[j]+usr->vals_tv[j] == 0 ) continue; // skip empty bins
float val = usr->min + (usr->max - usr->min)*j/(usr->nbins-1);
- const char *fmt = usr->type==BCF_HT_REAL ? "USR:%s/%d\t%d\t%e\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n" : "USR:%s/%d\t%d\t%.0f\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n";
+ const char * const fmt = usr->type==BCF_HT_REAL ? "USR:%s/%d\t%d\t%e\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n" : "USR:%s/%d\t%d\t%.0f\t%"PRIu64"\t%"PRIu64"\t%"PRIu64"\n";
fprintf(bcftools_stdout, fmt,usr->tag,usr->idx,id,val,usr->vals_ts[j]+usr->vals_tv[j],usr->vals_ts[j],usr->vals_tv[j]);
}
}
}
}
+ fprintf(bcftools_stdout, "# DP, depth:\n"
+ "# - set id, see above\n"
+ "# - the depth bin, corresponds to the depth (unless --depth was given)\n"
+ "# - number of genotypes with this depth (zero unless -s/-S was given)\n"
+ "# - fraction of genotypes with this depth (zero unless -s/-S was given)\n"
+ "# - number of sites with this depth\n"
+ "# - fraction of sites with this depth\n");
fprintf(bcftools_stdout, "# DP, Depth distribution\n# DP\t[2]id\t[3]bin\t[4]number of genotypes\t[5]fraction of genotypes (%%)\t[6]number of sites\t[7]fraction of sites (%%)\n");
for (id=0; id<args->nstats; id++)
{
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
+#include <htslib/kbitset.h>
#include "bcftools.h"
#include "filter.h"
#include "htslib/khash_str2int.h"
int record_cmd_line;
char *index_fn;
int write_index;
+ int trim_star_allele;
htsFile *out;
}
args_t;
int ret = bcf_trim_alleles(args->hsub ? args->hsub : args->hdr, line);
if ( ret<0 ) error("Error: Could not trim alleles at %s:%"PRId64"\n", bcf_seqname(args->hsub ? args->hsub : args->hdr, line), (int64_t) line->pos+1);
}
+ if (args->trim_star_allele)
+ {
+ int iunseen;
+ if ( args->trim_star_allele && (line->n_allele > 2 || args->trim_star_allele > 1) && (iunseen=get_unseen_allele(line)) && iunseen>0 )
+ {
+ // the unobserved star allele should be trimmed, either it is variant site or trimming of all sites was requested
+ kbitset_t *rm_set = kbs_init(line->n_allele);
+ kbs_insert(rm_set, iunseen);
+ if ( bcf_remove_allele_set(args->hdr,line,rm_set) )
+ error("[%s] Error: failed to trim the unobserved allele at %s:%"PRIhts_pos"\n",__func__,bcf_seqname(args->hdr,line),line->pos+1);
+ kbs_destroy(rm_set);
+ }
+ }
if (args->phased) {
int phased = bcf_all_phased(args->hdr, line);
if (args->phased == FLT_INCLUDE && !phased) { return 0; } // skip unphased
fprintf(stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
fprintf(stderr, "\n");
fprintf(stderr, "Subset options:\n");
+ fprintf(stderr, " -A, --trim-unseen-allele Remove '<*>' or '<NON_REF>' at variant (-A) or at all (-AA) sites\n");
fprintf(stderr, " -a, --trim-alt-alleles Trim ALT alleles not seen in the genotype fields (or their subset with -s/-S)\n");
fprintf(stderr, " -I, --no-update Do not (re)calculate INFO fields for the subset (currently INFO/AC and INFO/AN)\n");
fprintf(stderr, " -s, --samples [^]LIST Comma separated list of samples to include (or exclude with \"^\" prefix). Be careful\n");
fprintf(stderr, " -u/U, --uncalled/--exclude-uncalled Select/exclude sites without a called genotype\n");
fprintf(stderr, " -v/V, --types/--exclude-types LIST Select/exclude comma-separated list of variant types: snps,indels,mnps,ref,bnd,other [null]\n");
fprintf(stderr, " -x/X, --private/--exclude-private Select/exclude sites where the non-reference alleles are exclusive (private) to the subset samples\n");
- fprintf(stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(stderr, "\n");
exit(1);
}
{"exclude",required_argument,NULL,'e'},
{"include",required_argument,NULL,'i'},
{"trim-alt-alleles",no_argument,NULL,'a'},
+ {"trim-unseen-allele",no_argument,NULL,'A'},
{"no-update",no_argument,NULL,'I'},
{"drop-genotypes",no_argument,NULL,'G'},
{"private",no_argument,NULL,'x'},
{"phased",no_argument,NULL,'p'},
{"exclude-phased",no_argument,NULL,'P'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "l:t:T:r:R:o:O:s:S:Gf:knv:V:m:M:auUhHc:C:Ii:e:xXpPq:Q:g:",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "l:t:T:r:R:o:O:s:S:Gf:knv:V:m:M:aAuUhHc:C:Ii:e:xXpPq:Q:g:W::",loptions,NULL)) >= 0)
{
char allele_type[9] = "nref";
switch (c)
case 'S': args->sample_names = optarg; args->sample_is_file = 1; break;
case 1 : args->force_samples = 1; break;
case 'a': args->trim_alts = 1; args->calc_ac = 1; break;
+ case 'A': args->trim_star_allele++; break;
case 'I': args->update_info = 0; break;
case 'G': args->sites_only = 1; break;
break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
- case 10 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
}
else if ( args->output_type & FT_BCF )
error("BCF output requires header, cannot proceed with -H\n");
- if ( args->write_index && init_index(args->out,out_hdr,args->fn_out,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->fn_out);
+ if ( init_index2(args->out,out_hdr,args->fn_out, &args->index_fn,
+ args->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",args->fn_out);
int ret = 0;
if (!args->header_only)
#include <htslib/vcf.h>
#include <htslib/synced_bcf_reader.h>
#include <htslib/vcfutils.h>
+#include <htslib/kbitset.h>
#include "bcftools.h"
#include "filter.h"
#include "htslib/khash_str2int.h"
int record_cmd_line;
char *index_fn;
int write_index;
+ int trim_star_allele;
htsFile *out;
}
args_t;
int ret = bcf_trim_alleles(args->hsub ? args->hsub : args->hdr, line);
if ( ret<0 ) error("Error: Could not trim alleles at %s:%"PRId64"\n", bcf_seqname(args->hsub ? args->hsub : args->hdr, line), (int64_t) line->pos+1);
}
+ if (args->trim_star_allele)
+ {
+ int iunseen;
+ if ( args->trim_star_allele && (line->n_allele > 2 || args->trim_star_allele > 1) && (iunseen=get_unseen_allele(line)) && iunseen>0 )
+ {
+ // the unobserved star allele should be trimmed, either it is variant site or trimming of all sites was requested
+ kbitset_t *rm_set = kbs_init(line->n_allele);
+ kbs_insert(rm_set, iunseen);
+ if ( bcf_remove_allele_set(args->hdr,line,rm_set) )
+ error("[%s] Error: failed to trim the unobserved allele at %s:%"PRIhts_pos"\n",__func__,bcf_seqname(args->hdr,line),line->pos+1);
+ kbs_destroy(rm_set);
+ }
+ }
if (args->phased) {
int phased = bcf_all_phased(args->hdr, line);
if (args->phased == FLT_INCLUDE && !phased) { return 0; } // skip unphased
fprintf(bcftools_stderr, " --threads INT Use multithreading with INT worker threads [0]\n");
fprintf(bcftools_stderr, "\n");
fprintf(bcftools_stderr, "Subset options:\n");
+ fprintf(bcftools_stderr, " -A, --trim-unseen-allele Remove '<*>' or '<NON_REF>' at variant (-A) or at all (-AA) sites\n");
fprintf(bcftools_stderr, " -a, --trim-alt-alleles Trim ALT alleles not seen in the genotype fields (or their subset with -s/-S)\n");
fprintf(bcftools_stderr, " -I, --no-update Do not (re)calculate INFO fields for the subset (currently INFO/AC and INFO/AN)\n");
fprintf(bcftools_stderr, " -s, --samples [^]LIST Comma separated list of samples to include (or exclude with \"^\" prefix). Be careful\n");
fprintf(bcftools_stderr, " -u/U, --uncalled/--exclude-uncalled Select/exclude sites without a called genotype\n");
fprintf(bcftools_stderr, " -v/V, --types/--exclude-types LIST Select/exclude comma-separated list of variant types: snps,indels,mnps,ref,bnd,other [null]\n");
fprintf(bcftools_stderr, " -x/X, --private/--exclude-private Select/exclude sites where the non-reference alleles are exclusive (private) to the subset samples\n");
- fprintf(bcftools_stderr, " --write-index Automatically index the output files [off]\n");
+ fprintf(bcftools_stderr, " -W, --write-index[=FMT] Automatically index the output files [off]\n");
fprintf(bcftools_stderr, "\n");
bcftools_exit(1);
}
{"exclude",required_argument,NULL,'e'},
{"include",required_argument,NULL,'i'},
{"trim-alt-alleles",no_argument,NULL,'a'},
+ {"trim-unseen-allele",no_argument,NULL,'A'},
{"no-update",no_argument,NULL,'I'},
{"drop-genotypes",no_argument,NULL,'G'},
{"private",no_argument,NULL,'x'},
{"phased",no_argument,NULL,'p'},
{"exclude-phased",no_argument,NULL,'P'},
{"no-version",no_argument,NULL,8},
- {"write-index",no_argument,NULL,10},
+ {"write-index",optional_argument,NULL,'W'},
{NULL,0,NULL,0}
};
char *tmp;
- while ((c = getopt_long(argc, argv, "l:t:T:r:R:o:O:s:S:Gf:knv:V:m:M:auUhHc:C:Ii:e:xXpPq:Q:g:",loptions,NULL)) >= 0)
+ while ((c = getopt_long(argc, argv, "l:t:T:r:R:o:O:s:S:Gf:knv:V:m:M:aAuUhHc:C:Ii:e:xXpPq:Q:g:W::",loptions,NULL)) >= 0)
{
char allele_type[9] = "nref";
switch (c)
case 'S': args->sample_names = optarg; args->sample_is_file = 1; break;
case 1 : args->force_samples = 1; break;
case 'a': args->trim_alts = 1; args->calc_ac = 1; break;
+ case 'A': args->trim_star_allele++; break;
case 'I': args->update_info = 0; break;
case 'G': args->sites_only = 1; break;
break;
case 9 : args->n_threads = strtol(optarg, 0, 0); break;
case 8 : args->record_cmd_line = 0; break;
- case 10 : args->write_index = 1; break;
+ case 'W':
+ if (!(args->write_index = write_index_parse(optarg)))
+ error("Unsupported index format '%s'\n", optarg);
+ break;
case '?': usage(args); break;
default: error("Unknown argument: %s\n", optarg);
}
else if ( args->output_type & FT_BCF )
error("BCF output requires header, cannot proceed with -H\n");
- if ( args->write_index && init_index(args->out,out_hdr,args->fn_out,&args->index_fn)<0 ) error("Error: failed to initialise index for %s\n",args->fn_out);
+ if ( init_index2(args->out,out_hdr,args->fn_out, &args->index_fn,
+ args->write_index) < 0 )
+ error("Error: failed to initialise index for %s\n",args->fn_out);
int ret = 0;
if (!args->header_only)
/* version.c -- report version numbers for plugins.
- Copyright (C) 2014-2023 Genome Research Ltd.
+ Copyright (C) 2014-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
const char *end = fname ? strstr(fname, HTS_IDX_DELIM) : NULL;
if ( !end ) end = fname ? fname + strlen(fname) : fname;
int len = end - fname;
- if ( len >= 4 && !strncasecmp(".bcf",fname+len-4,4) ) ret = hts_bcf_wmode(FT_BCF|FT_GZ);
+ if ( len >= 4 && !strncasecmp(".bcf",fname+len-4,4) )
+ ret = hts_bcf_wmode(file_type & FT_BCF ? file_type : FT_BCF|FT_GZ);
else if ( len >= 4 && !strncasecmp(".vcf",fname+len-4,4) ) ret = hts_bcf_wmode(FT_VCF);
else if ( len >= 7 && !strncasecmp(".vcf.gz",fname+len-7,7) ) ret = hts_bcf_wmode(FT_VCF|FT_GZ);
else if ( len >= 8 && !strncasecmp(".vcf.bgz",fname+len-8,8) ) ret = hts_bcf_wmode(FT_VCF|FT_GZ);
else return -1;
}
+// Used to set args->write_index in CLI.
+// It will be true if set correctly.
+// Note due to HTS_FMT_CSI being zero we have to use an additional bit.
+int write_index_parse(char *arg) {
+ int fmt = HTS_FMT_CSI;
+
+ if (arg) {
+ if (strcmp(arg, "csi") == 0 || strcmp(arg, "=csi") == 0)
+ fmt = HTS_FMT_CSI;
+ else if (strcmp(arg, "tbi") == 0 || strcmp(arg, "=tbi") == 0)
+ fmt = HTS_FMT_TBI;
+ else
+ return 0;
+ }
+
+ return 128 | fmt;
+}
+
// See also samtools/sam_utils.c auto_index()
-int init_index(htsFile *fh, bcf_hdr_t *hdr, char *fname, char **idx_fname)
-{
- int min_shift = 14; // CSI
+int init_index2(htsFile *fh, bcf_hdr_t *hdr, const char *fname,
+ char **idx_fname, int idx_fmt) {
+ // Nothing to do == success. This simplifies the main code simpler.
+ if (!idx_fmt)
+ return 0;
+
+ int min_shift;
+ char *idx_suffix;
+
+ if (idx_fmt && (idx_fmt&127) == HTS_FMT_TBI && fh->format.format == vcf) {
+ min_shift = 0; // TBI
+ idx_suffix = "tbi";
+ } else {
+ min_shift = 14; // CSI
+ idx_suffix = "csi";
+ }
if ( !fname || !*fname || !strcmp(fname, "-") ) return -1;
char *delim = strstr(fname, HTS_IDX_DELIM);
- if (delim)
- {
+ if (delim) {
delim += strlen(HTS_IDX_DELIM);
*idx_fname = strdup(delim);
if ( !*idx_fname ) return -1;
size_t l = strlen(*idx_fname);
if ( l >= 4 && strcmp(*idx_fname + l - 4, ".tbi")==0 ) min_shift = 0;
- }
- else
- {
+ } else {
if ( !(*idx_fname = malloc(strlen(fname)+6)) ) return -1;
- sprintf(*idx_fname, "%s.csi", fname);
+ sprintf(*idx_fname, "%s.%s", fname, idx_suffix);
}
if ( bcf_idx_init(fh, hdr, min_shift, *idx_fname) < 0 ) return -1;
return 0;
}
-
+int init_index(htsFile *fh, bcf_hdr_t *hdr, const char *fname, char **idx_fname)
+{
+ return init_index2(fh,hdr, fname, idx_fname, HTS_FMT_CSI);
+}
/* version.c -- report version numbers for plugins.
- Copyright (C) 2014-2023 Genome Research Ltd.
+ Copyright (C) 2014-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
const char *end = fname ? strstr(fname, HTS_IDX_DELIM) : NULL;
if ( !end ) end = fname ? fname + strlen(fname) : fname;
int len = end - fname;
- if ( len >= 4 && !strncasecmp(".bcf",fname+len-4,4) ) ret = hts_bcf_wmode(FT_BCF|FT_GZ);
+ if ( len >= 4 && !strncasecmp(".bcf",fname+len-4,4) )
+ ret = hts_bcf_wmode(file_type & FT_BCF ? file_type : FT_BCF|FT_GZ);
else if ( len >= 4 && !strncasecmp(".vcf",fname+len-4,4) ) ret = hts_bcf_wmode(FT_VCF);
else if ( len >= 7 && !strncasecmp(".vcf.gz",fname+len-7,7) ) ret = hts_bcf_wmode(FT_VCF|FT_GZ);
else if ( len >= 8 && !strncasecmp(".vcf.bgz",fname+len-8,8) ) ret = hts_bcf_wmode(FT_VCF|FT_GZ);
else return -1;
}
+// Used to set args->write_index in CLI.
+// It will be true if set correctly.
+// Note due to HTS_FMT_CSI being zero we have to use an additional bit.
+int write_index_parse(char *arg) {
+ int fmt = HTS_FMT_CSI;
+
+ if (arg) {
+ if (strcmp(arg, "csi") == 0 || strcmp(arg, "=csi") == 0)
+ fmt = HTS_FMT_CSI;
+ else if (strcmp(arg, "tbi") == 0 || strcmp(arg, "=tbi") == 0)
+ fmt = HTS_FMT_TBI;
+ else
+ return 0;
+ }
+
+ return 128 | fmt;
+}
+
// See also samtools/sam_utils.c auto_index()
-int init_index(htsFile *fh, bcf_hdr_t *hdr, char *fname, char **idx_fname)
-{
- int min_shift = 14; // CSI
+int init_index2(htsFile *fh, bcf_hdr_t *hdr, const char *fname,
+ char **idx_fname, int idx_fmt) {
+ // Nothing to do == success. This simplifies the main code simpler.
+ if (!idx_fmt)
+ return 0;
+
+ int min_shift;
+ char *idx_suffix;
+
+ if (idx_fmt && (idx_fmt&127) == HTS_FMT_TBI && fh->format.format == vcf) {
+ min_shift = 0; // TBI
+ idx_suffix = "tbi";
+ } else {
+ min_shift = 14; // CSI
+ idx_suffix = "csi";
+ }
if ( !fname || !*fname || !strcmp(fname, "-") ) return -1;
char *delim = strstr(fname, HTS_IDX_DELIM);
- if (delim)
- {
+ if (delim) {
delim += strlen(HTS_IDX_DELIM);
*idx_fname = strdup(delim);
if ( !*idx_fname ) return -1;
size_t l = strlen(*idx_fname);
if ( l >= 4 && strcmp(*idx_fname + l - 4, ".tbi")==0 ) min_shift = 0;
- }
- else
- {
+ } else {
if ( !(*idx_fname = malloc(strlen(fname)+6)) ) return -1;
- sprintf(*idx_fname, "%s.csi", fname);
+ sprintf(*idx_fname, "%s.%s", fname, idx_suffix);
}
if ( bcf_idx_init(fh, hdr, min_shift, *idx_fname) < 0 ) return -1;
return 0;
}
-
+int init_index(htsFile *fh, bcf_hdr_t *hdr, const char *fname, char **idx_fname)
+{
+ return init_index2(fh,hdr, fname, idx_fname, HTS_FMT_CSI);
+}
# DEALINGS IN THE SOFTWARE.
# Master version, for use in tarballs or non-git source copies
-VERSION=1.18
+VERSION=1.21
# If we have a git clone, then check against the current tag
if [ -e .git ]
--- /dev/null
+#!/bin/sh -e
+
+case $1 in
+ ubuntu-*-arm) expected=arm ;;
+ macos-13) expected=x86_64 ;;
+ ubuntu-*) expected=x86_64 ;;
+ macos-*) expected=arm ;;
+ windows-*) expected=x86_64 ;;
+ *)
+ echo Unknown platform $1 >&2
+ exit 2
+ ;;
+esac
+
+arch=$(uname -m)
+case $arch in
+ arm*|aarch*) actual=arm ;;
+ x86*) actual=x86_64 ;;
+ *)
+ echo Unrecognized uname result $arch >&2
+ exit 2
+ ;;
+esac
+
+if test $actual = $expected
+then
+ echo Running on $arch as expected
+ exit 0
+else
+ echo Platform $arch is not the expected $expected >&2
+ exit 1
+fi
- conda-forge
- bioconda
dependencies:
+ - bzip2
- cython
- setuptools
+ - xz
import fnmatch
import os
+from pathlib import Path
import re
import itertools
import shutil
"peakfit.h",
"polysomy.c"),
"htslib": (
- 'htslib/tabix.c', 'htslib/bgzip.c',
- 'htslib/htsfile.c',
+ 'annot-tsv.c', 'bgzip.c', 'htsfile.c', 'tabix.c',
+ 'hts_probe_cc.sh',
"samples", "test", "tests"),
}
if fn == "bamtk.c":
lines = re.sub(r'(#include "version.h")', r'\1\n#include "samtools_config_vars.h"', lines)
lines = re.sub(r'(else if.*"tview")', r'//\1', lines)
+ lines = re.sub(r'(if[ (]*fclose)', r'if (0) { //\1', lines)
outfile.write(lines)
locate("version.sh", srcdir, exclude_htslib=True))
if dest == "htslib":
- # Add build files, including *.ac *.in *.mk *.m4 *.sh
+ # Add build files, including *.ac config.{guess,sub} *.in *.mk *.m4 *.sh
mfiles = itertools.chain(mfiles, locate("Makefile", srcdir),
locate("configure", srcdir),
+ locate("config.*", srcdir),
locate("*.[aims][cnk4h]", srcdir, exclude))
ncopied = 0
_update_version_file(C_VERSION[dest], version + " (pysam)", "pysam/version.h")
_update_version_doc_file(dest, version, "README.rst")
_update_version_doc_file(dest, version, "doc/index.rst")
+ if dest in MAIN:
+ path = Path("doc/conf.py")
+ path.write_text(re.sub(rf"doc/[\d.]*/{dest}", rf"doc/{version}/{dest}", path.read_text()))
sys.exit(0)
# General information about the project.
project = u'pysam'
-copyright = '2009–2024 Andreas Heger, John Marshall, Kevin Jacobs, et al'
+copyright = '2009–2025 Andreas Heger, John Marshall, Kevin Jacobs, et al'
# Included at the end of each rst file
rst_epilog = '''
-.. _CGAT Training Programme: http://www.cgat.org
.. _pysam: https://github.com/pysam-developers/pysam
-.. _samtools: http://samtools.sourceforge.net/
-.. _bcftools: https://samtools.github.io/bcftools/bcftools.html
-.. _htslib: http://www.htslib.org/
-.. _tabix: http://www.htslib.org/doc/tabix.html
-.. _Galaxy: https://main.g2.bx.psu.edu/
+.. _samtools: https://www.htslib.org/doc/1.21/samtools.html
+.. _bcftools: https://www.htslib.org/doc/1.21/bcftools.html
+.. _htslib: https://www.htslib.org/
+.. _tabix: https://www.htslib.org/doc/tabix.html
+.. _Galaxy: https://usegalaxy.org/
.. _cython: https://cython.org/
.. _python: https://www.python.org/
-.. _pypi: https://pypi.org/
+.. _PyPI: https://pypi.org/
.. _pip: https://pip.pypa.io/
.. _pyximport: https://github.com/cython/cython/tree/master/pyximport
.. _conda: https://conda.io/docs/
:file:`import.py` about importing.
+Python language level
+=====================
+
+Pysam currently requires Python 3.6 as a minimum language level.
+For example, this means that the following comparatively recent
+language features and library functions are available for use:
+
+* f-strings
+* ``raise ... from None``
+* :meth:`str.startswith`, :meth:`str.endswith`, :meth:`str.rstrip`, etc
+
+However in particular the the following should not be used in
+pysam source code or infrastructure scripts:
+
+* :meth:`str.removeprefix`, :meth:`str.removesuffix` (new in 3.9)
+* walrus ``:=`` operator (new in 3.8)
+
+
Importing new versions of htslib and samtools
=============================================
AlignmentFile.fetch does not show unmapped reads
================================================
-:meth:`~pysam.AlignmentFile.fetch` will only iterate over alignments
-in the SAM/BAM file. The following thus always works::
+By default, :meth:`~pysam.AlignmentFile.fetch` will only iterate over
+placed alignments in the SAM/BAM/CRAM file. Thus the following always
+works::
- bf = pysam.AlignmentFile(fname, "rb")
- for r in bf.fetch():
- assert not r.is_unmapped
+ f = pysam.AlignmentFile(fname, "r")
+ for r in f.fetch():
+ assert r.reference_name is not None
-If the SAM/BAM file contains unaligned reads, they can be included
+If the file contains unaligned reads, they can be included
in the iteration by adding the ``until_eof=True`` flag::
- bf = pysam.AlignmentFile(fname, "rb")
- for r in bf.fetch(until_eof=True):
+ f = pysam.AlignmentFile(fname, "r")
+ for r in f.fetch(until_eof=True):
if r.is_unmapped:
- print("read is unmapped")
+ print("read is unmapped")
+
+See also :meth:`fetch("*") <pysam.AlignmentFile.fetch>` which iterates
+only over the unplaced unmapped reads at the end of the file.
I can't call AlignmentFile.fetch on a file without an index
===========================================================
using cython and a high-level, pythonic API for convenient access to
the data within genomic file formats.
-The current version wraps *htslib-1.18*, *samtools-1.18*, and *bcftools-1.18*.
+The current version wraps *htslib-1.21*, *samtools-1.21*, and *bcftools-1.21*.
To install the latest release, type::
Installing pysam
================
-Pysam can be installed through conda_, pypi_ and from the repository.
+Pysam can be installed through conda_, PyPI_ and from the repository.
The recommended way to install pysam is through conda/bioconda.
Conda installation
will be set automatically, which will potentially save a lot of
trouble on OS X.
-Pypi installation
+PyPI installation
=================
Pysam provides a python interface to the functionality contained
Builtin
-------
-The typical installation will be through pypi_::
+The typical installation will be through PyPI_::
pip install pysam
python setup.py install
-For compilation options, see the section on Pypi installation above.
+For compilation options, see the section on PyPI installation above.
Requirements
============
Note that the file open mode needs to changed from ``r`` to ``rb``.
-=====================================
-Using samtools commands within python
-=====================================
+==================================================
+Using samtools and bcftools commands within Python
+==================================================
-Commands available in `samtools`_ are available as simple
-function calls. Command line options are provided as arguments. For
+Commands available in `samtools`_ and `bcftools`_ are available as simple
+function calls, with command line options provided as arguments. For
example::
- pysam.sort("-o", "output.bam", "ex1.bam")
+ import pysam.samtools
+ pysam.samtools.sort("-o", "output.bam", "ex1.bam", catch_stdout=False)
-corresponds to the command line::
+ import pysam.bcftools
+ pysam.bcftools.index("--csi", "ex2.vcf.gz")
+
+corresponds to the command lines::
samtools sort -o output.bam ex1.bam
+ bcftools index --csi ex2.vcf.gz
+
+Samtools commands are also imported into the main ``pysam`` namespace.
+For example::
-Or for example::
+ pysam.sort("-m", "1000000", "-o", "output.bam", "ex1.bam", catch_stdout=False)
- pysam.sort("-m", "1000000", "-o", "output.bam", "ex1.bam")
+To make them valid Python identifiers, the functions :func:`!cram_size`
+and :func:`!fqimport` are spelt thus, differently from their
+corresponding commands.
In order to get usage information, try::
pysam.SamtoolsError: 'Usage: samtools sort [-n] [-m <maxMem>] <in.bam> <out.prefix>\n'
Messages from `samtools`_ on stderr are captured and are
-available using the :meth:`getMessages` method::
+available using the :meth:`~PysamDispatcher.get_messages` method::
+
+ pysam.sort.get_messages()
+
+By default, pysam captures the samtools command's standard output and returns it
+as the function's return value. To redirect stdout to a file instead, either use
+the ``save_stdout`` keyword argument, or use ``"-o", "filename"`` in the arguments
+and also use ``catch_stdout=False`` to prevent pysam's capturing from overriding
+your redirection. Finally, ``catch_stdout=False`` by itself discards standard output,
+which may help resolve problems in environments such as IPython notebooks::
+
+ # Return value
+ pileup_text = pysam.samtools.mpileup("in.bam")
+
+ # Save to file
+ pysam.samtools.mpileup("in.bam", save_stdout=pileup_filename)
+ pysam.samtools.mpileup("-o", pileup_filename, "in.bam", catch_stdout=False)
+
+ # Discard standard output
+ pysam.samtools.mpileup("in.bam", catch_stdout=False) # Returns None
+
+For each :obj:`!command` available as a `samtools`_ subcommand,
+the following functions are provided:
+
+.. py:function:: pysam.samtools.command(args, *, catch_stdout=True, save_stdout=None, split_lines=False)
+
+ :param args: Arguments to be passed to the samtools subcommand.
+ :param bool catch_stdout: Whether to return stdout as the function's value.
+ :param str save_stdout: Filename to which stdout should be written.
+ :param bool split_lines: Whether to split the return value into a list of lines.
+ :returns: Standard output if it was caught, otherwise None.
+
+ If `save_stdout` is not None, the command's standard ouput is written to the
+ file specified and the function returns None.
+
+ Otherwise, if `catch_stdout` is true, the command's standard output is captured
+ and used as the function's return value --- either as a single :obj:`str` or as
+ :obj:`list[str] <list>` according to `split_lines`. If `catch_stdout` is false,
+ the command's standard output is discarded and the function returns None.
+
+ The command's standard error is always captured and made available via
+ :func:`~pysam.samtools.command.get_messages`.
+
+.. py:function:: pysam.samtools.command.get_messages()
- pysam.sort.getMessage()
+ Returns the standard error from the most recent invocation of the particular
+ :obj:`!command`, either as a single :obj:`str` or as :obj:`list[str] <list>`
+ according to `split_lines` as specified in that invocation.
-Note that only the output from the last invocation of a command is
-stored.
+.. py:function:: pysam.samtools.command.usage()
-In order for pysam to make the output of samtools commands accessible
-the stdout stream needs to be redirected. This is the default
-behaviour, but can cause problems in environments such as the ipython
-notebook. A solution is to pass the ``catch_stdout`` keyword
-argument::
+ Returns the command's usage/help message, as a single :obj:`str`.
- pysam.sort(catch_stdout=False)
+For each :obj:`!command` available as a `bcftools`_ subcommand, the
+:func:`!pysam.bcftools.command`, :func:`!pysam.bcftools.command.get_messages`,
+and :func:`!pysam.bcftools.command.usage` functions operate similarly.
-Note that this means that output from commands which produce output on
-stdout will not be available. The only solution is to run samtools
-commands through subprocess.
================================
Working with tabix-indexed files
-try:
- from typing import Final
- HAVE_FINAL = True
-except ImportError:
- HAVE_FINAL = False
+import pysam.utils
-from pysam.utils import PysamDispatcher
+annotate = pysam.utils.PysamDispatcher('bcftools', 'annotate')
+call = pysam.utils.PysamDispatcher('bcftools', 'call')
+cnv = pysam.utils.PysamDispatcher('bcftools', 'cnv')
+concat = pysam.utils.PysamDispatcher('bcftools', 'concat')
+consensus = pysam.utils.PysamDispatcher('bcftools', 'consensus')
+convert = pysam.utils.PysamDispatcher('bcftools', 'convert')
+csq = pysam.utils.PysamDispatcher('bcftools', 'csq')
+filter = pysam.utils.PysamDispatcher('bcftools', 'filter')
+gtcheck = pysam.utils.PysamDispatcher('bcftools', 'gtcheck')
+head = pysam.utils.PysamDispatcher('bcftools', 'head')
+index = pysam.utils.PysamDispatcher('bcftools', 'index')
+isec = pysam.utils.PysamDispatcher('bcftools', 'isec')
+merge = pysam.utils.PysamDispatcher('bcftools', 'merge')
+mpileup = pysam.utils.PysamDispatcher('bcftools', 'mpileup')
+norm = pysam.utils.PysamDispatcher('bcftools', 'norm')
+plugin = pysam.utils.PysamDispatcher('bcftools', 'plugin')
+query = pysam.utils.PysamDispatcher('bcftools', 'query')
+reheader = pysam.utils.PysamDispatcher('bcftools', 'reheader')
+roh = pysam.utils.PysamDispatcher('bcftools', 'roh')
+sort = pysam.utils.PysamDispatcher('bcftools', 'sort')
+stats = pysam.utils.PysamDispatcher('bcftools', 'stats')
+view = pysam.utils.PysamDispatcher('bcftools', 'view')
-_BCFTOOLS_DISPATCH = [
- "index",
- "annotate",
- "concat",
- "convert",
- "isec",
- "merge",
- "norm",
- "plugin",
- "query",
- "reheader",
- "sort",
- "view",
- "head",
- "call",
- "consensus",
- "cnv",
- "csq",
- "filter",
- "gtcheck",
- "mpileup",
- "roh",
- "stats"]
-
-
-def _wrap_command(dispatch: str) -> PysamDispatcher:
- return PysamDispatcher("bcftools", dispatch, ())
-
-
-if not HAVE_FINAL:
- # instantiate bcftools commands as python functions
- for cmd in _BCFTOOLS_DISPATCH:
- globals()[cmd] = PysamDispatcher("bcftools", cmd, None)
-else:
- # python >=3.8
- index: Final[PysamDispatcher] = _wrap_command("index")
- annotate: Final[PysamDispatcher] = _wrap_command("annotate")
- concat: Final[PysamDispatcher] = _wrap_command("concat")
- convert: Final[PysamDispatcher] = _wrap_command("convert")
- isec: Final[PysamDispatcher] = _wrap_command("isec")
- merge: Final[PysamDispatcher] = _wrap_command("merge")
- norm: Final[PysamDispatcher] = _wrap_command("norm")
- plugin: Final[PysamDispatcher] = _wrap_command("plugin")
- query: Final[PysamDispatcher] = _wrap_command("query")
- reheader: Final[PysamDispatcher] = _wrap_command("reheader")
- sort: Final[PysamDispatcher] = _wrap_command("sort")
- view: Final[PysamDispatcher] = _wrap_command("view")
- head: Final[PysamDispatcher] = _wrap_command("head")
- call: Final[PysamDispatcher] = _wrap_command("call")
- consensus: Final[PysamDispatcher] = _wrap_command("consensus")
- cnv: Final[PysamDispatcher] = _wrap_command("cnv")
- csq: Final[PysamDispatcher] = _wrap_command("csq")
- filter: Final[PysamDispatcher] = _wrap_command("filter")
- gtcheck: Final[PysamDispatcher] = _wrap_command("gtcheck")
- mpileup: Final[PysamDispatcher] = _wrap_command("mpileup")
- roh: Final[PysamDispatcher] = _wrap_command("roh")
- stats: Final[PysamDispatcher] = _wrap_command("stats")
+__all__ = [
+ 'annotate', 'call', 'cnv', 'concat', 'consensus',
+ 'convert', 'csq', 'filter', 'gtcheck', 'head',
+ 'index', 'isec', 'merge', 'mpileup', 'norm',
+ 'plugin', 'query', 'reheader', 'roh', 'sort',
+ 'stats', 'view',
+]
TagValue = Union[str, int, float, array]
class CIGAR_OPS(enum.IntEnum):
- CBACK: int
- CDEL: int
- CDIFF: int
- CEQUAL: int
- CHARD_CLIP: int
- CINS: int
- CMATCH: int
- CPAD: int
- CREF_SKIP: int
- CSOFT_CLIP: int
+ CBACK = ...
+ CDEL = ...
+ CDIFF = ...
+ CEQUAL = ...
+ CHARD_CLIP = ...
+ CINS = ...
+ CMATCH = ...
+ CPAD = ...
+ CREF_SKIP = ...
+ CSOFT_CLIP = ...
class SAM_FLAGS(enum.IntEnum):
- FDUP: int
- FMREVERSE: int
- FMUNMAP: int
- FPAIRED: int
- FPROPER_PAIR: int
- FQCFAIL: int
- FREAD1: int
- FREAD2: int
- FREVERSE: int
- FSECONDARY: int
- FSUPPLEMENTARY: int
- FUNMAP: int
+ FDUP = ...
+ FMREVERSE = ...
+ FMUNMAP = ...
+ FPAIRED = ...
+ FPROPER_PAIR = ...
+ FQCFAIL = ...
+ FREAD1 = ...
+ FREAD2 = ...
+ FREVERSE = ...
+ FSECONDARY = ...
+ FSUPPLEMENTARY = ...
+ FUNMAP = ...
class AlignedSegment:
header: AlignmentHeader
def get_forward_sequence(self) -> Optional[str]: ...
def get_forward_qualities(self) -> Optional[array]: ...
def get_aligned_pairs(
- self, matches_only: bool = ..., with_seq: bool = ...
+ self, matches_only: bool = ..., with_seq: bool = ..., with_cigar: bool = ...
) -> List[Tuple[int, int]]: ...
def get_blocks(self) -> List[Tuple[int, int]]: ...
def get_overlap(self, start: int, end: int) -> Optional[int]: ...
raise ValueError("unsupported type code '{}'".format(value.typecode))
if typecode not in DATATYPE2FORMAT:
- raise ValueError("invalid value type '{}' ({})".format(chr(typecode), array.typecode))
+ raise ValueError("invalid value type '{}'".format(chr(typecode)))
# use array.tostring() to retrieve byte representation and
# save as bytes
if typecode == b'Z' or typecode == b'H':
datafmt = "2sB%is" % (len(value)+1)
- else:
+ elif typecode in DATATYPE2FORMAT:
datafmt = "2sB%s" % DATATYPE2FORMAT[typecode][0]
+ else:
+ raise ValueError("invalid value type '{}'".format(chr(typecode)))
args.extend([pytag[:2],
typecode,
elif op == BAM_CHARD_CLIP:
pass # advances neither
- cdef char *md_tag, md_buffer[2];
+ cdef char md_buffer[2]
+ cdef char *md_tag
cdef uint8_t md_typecode = md_tag_ptr[0]
if md_typecode == b'Z':
md_tag = bam_aux2Z(md_tag_ptr)
return seq
+cdef inline str safe_reference_name(AlignmentHeader header, int tid):
+ if tid == -1: return "*"
+ elif header is not None: return header.get_reference_name(tid)
+ else: return f"#{tid}"
+
+
+# Tuple-building helper functions used by AlignedSegment.get_aligned_pairs()
+
+cdef _alignedpairs_positions(qpos, pos, ref_seq, uint32_t r_idx, int op):
+ return (qpos, pos)
+
+
+cdef _alignedpairs_with_seq(qpos, pos, ref_seq, uint32_t r_idx, int op):
+ ref_base = ref_seq[r_idx] if ref_seq is not None else None
+ return (qpos, pos, ref_base)
+
+
+cdef _alignedpairs_with_cigar(qpos, pos, ref_seq, uint32_t r_idx, int op):
+ return (qpos, pos, CIGAR_OPS(op))
+
+
+cdef _alignedpairs_with_seq_cigar(qpos, pos, ref_seq, uint32_t r_idx, int op):
+ ref_base = ref_seq[r_idx] if ref_seq is not None else None
+ return (qpos, pos, ref_base, CIGAR_OPS(op))
+
+
cdef class AlignedSegment:
'''Class representing an aligned segment.
The representation is an approximate :term:`SAM` format, because
an aligned read might not be associated with a :term:`AlignmentFile`.
- As a result :term:`tid` is shown instead of the reference name.
+ Hence when the read does not have an associated :class:`AlignedHeader`,
+ :term:`tid` is shown instead of the reference name.
Similarly, the tags field is returned in its parsed state.
To get a valid SAM record, use :meth:`to_string`.
# requires a valid header.
return "\t".join(map(str, (self.query_name,
self.flag,
- "#%d" % self.reference_id if self.reference_id >= 0 else "*",
+ safe_reference_name(self.header, self.reference_id),
self.reference_start + 1,
self.mapping_quality,
self.cigarstring,
- "#%d" % self.next_reference_id if self.next_reference_id >= 0 else "*",
+ safe_reference_name(self.header, self.next_reference_id),
self.next_reference_start + 1,
self.template_length,
self.query_sequence,
self.query_qualities,
self.tags)))
+ def __repr__(self):
+ ref = self.reference_name if self.header is not None else self.reference_id
+ return f'<{type(self).__name__}({self.query_name!r}, flags={self.flag}={self.flag:#x}, ref={ref!r}, zpos={self.reference_start}, mapq={self.mapping_quality}, cigar={self.cigarstring!r}, ...)>'
+
def __copy__(self):
return makeAlignedSegment(self._delegate, self.header)
else:
return self.query_qualities
-
- def get_aligned_pairs(self, matches_only=False, with_seq=False):
+ def get_aligned_pairs(self, matches_only=False, with_seq=False, with_cigar=False):
"""a list of aligned read (query) and reference positions.
Each item in the returned list is a tuple consisting of
reference sequence. For CIGAR 'P' (padding in the reference)
operations, the third tuple element will be None. Substitutions
are lower-case. This option requires an MD tag to be present.
+ with_cigar : bool
+ If True, return an extra element in the tuple containing the
+ CIGAR operator corresponding to this position tuple.
Returns
-------
cdef bam1_t * src = self._delegate
cdef bint _matches_only = bool(matches_only)
cdef bint _with_seq = bool(with_seq)
+ cdef bint _with_cigar = bool(with_cigar)
+ cdef object (*make_tuple)(object, object, object, uint32_t, int)
# TODO: this method performs no checking and assumes that
# read sequence, cigar and MD tag are consistent.
ref_seq = force_str(build_reference_sequence(src))
if ref_seq is None:
raise ValueError("MD tag not present")
+ make_tuple = _alignedpairs_with_seq_cigar if _with_cigar else _alignedpairs_with_seq
+ else:
+ ref_seq = None
+ make_tuple = _alignedpairs_with_cigar if _with_cigar else _alignedpairs_positions
r_idx = 0
l = cigar_p[k] >> BAM_CIGAR_SHIFT
if op == BAM_CMATCH or op == BAM_CEQUAL or op == BAM_CDIFF:
- if _with_seq:
- for i from pos <= i < pos + l:
- result.append((qpos, i, ref_seq[r_idx]))
- r_idx += 1
- qpos += 1
- else:
- for i from pos <= i < pos + l:
- result.append((qpos, i))
- qpos += 1
+ for i from pos <= i < pos + l:
+ result.append(make_tuple(qpos, i, ref_seq, r_idx, op))
+ r_idx += 1
+ qpos += 1
pos += l
elif op == BAM_CINS or op == BAM_CSOFT_CLIP or op == BAM_CPAD:
if not _matches_only:
- if _with_seq:
- for i from pos <= i < pos + l:
- result.append((qpos, None, None))
- qpos += 1
- else:
- for i from pos <= i < pos + l:
- result.append((qpos, None))
- qpos += 1
+ for i from pos <= i < pos + l:
+ result.append(make_tuple(qpos, None, None, 0, op))
+ qpos += 1
else:
qpos += l
elif op == BAM_CDEL:
if not _matches_only:
- if _with_seq:
- for i from pos <= i < pos + l:
- result.append((None, i, ref_seq[r_idx]))
- r_idx += 1
- else:
- for i from pos <= i < pos + l:
- result.append((None, i))
+ for i from pos <= i < pos + l:
+ result.append(make_tuple(None, i, ref_seq, r_idx, op))
+ r_idx += 1
else:
r_idx += l
pos += l
elif op == BAM_CREF_SKIP:
if not _matches_only:
- if _with_seq:
- for i from pos <= i < pos + l:
- result.append((None, i, None))
- else:
- for i from pos <= i < pos + l:
- result.append((None, i))
+ for i from pos <= i < pos + l:
+ result.append(make_tuple(None, i, None, 0, op))
pos += l
The output order in the array is "MIDNSHP=X" followed by a
field for the NM tag. If the NM tag is not present, this
- field will always be 0.
-
- +-----+--------------+-----+
- |M |BAM_CMATCH |0 |
- +-----+--------------+-----+
- |I |BAM_CINS |1 |
- +-----+--------------+-----+
- |D |BAM_CDEL |2 |
- +-----+--------------+-----+
- |N |BAM_CREF_SKIP |3 |
- +-----+--------------+-----+
- |S |BAM_CSOFT_CLIP|4 |
- +-----+--------------+-----+
- |H |BAM_CHARD_CLIP|5 |
- +-----+--------------+-----+
- |P |BAM_CPAD |6 |
- +-----+--------------+-----+
- |= |BAM_CEQUAL |7 |
- +-----+--------------+-----+
- |X |BAM_CDIFF |8 |
- +-----+--------------+-----+
- |B |BAM_CBACK |9 |
- +-----+--------------+-----+
- |NM |NM tag |10 |
- +-----+--------------+-----+
+ field will always be 0. (Accessing this field via index -1
+ avoids changes if more CIGAR operators are added in future.)
+
+ +-----+----------------+--------+
+ |M |pysam.CMATCH |0 |
+ +-----+----------------+--------+
+ |I |pysam.CINS |1 |
+ +-----+----------------+--------+
+ |D |pysam.CDEL |2 |
+ +-----+----------------+--------+
+ |N |pysam.CREF_SKIP |3 |
+ +-----+----------------+--------+
+ |S |pysam.CSOFT_CLIP|4 |
+ +-----+----------------+--------+
+ |H |pysam.CHARD_CLIP|5 |
+ +-----+----------------+--------+
+ |P |pysam.CPAD |6 |
+ +-----+----------------+--------+
+ |= |pysam.CEQUAL |7 |
+ +-----+----------------+--------+
+ |X |pysam.CDIFF |8 |
+ +-----+----------------+--------+
+ |B |pysam.CBACK |9 |
+ +-----+----------------+--------+
+ |NM |NM tag |10 or -1|
+ +-----+----------------+--------+
If no cigar string is present, empty arrays will be returned.
The operations are:
- +-----+--------------+-----+
- |M |BAM_CMATCH |0 |
- +-----+--------------+-----+
- |I |BAM_CINS |1 |
- +-----+--------------+-----+
- |D |BAM_CDEL |2 |
- +-----+--------------+-----+
- |N |BAM_CREF_SKIP |3 |
- +-----+--------------+-----+
- |S |BAM_CSOFT_CLIP|4 |
- +-----+--------------+-----+
- |H |BAM_CHARD_CLIP|5 |
- +-----+--------------+-----+
- |P |BAM_CPAD |6 |
- +-----+--------------+-----+
- |= |BAM_CEQUAL |7 |
- +-----+--------------+-----+
- |X |BAM_CDIFF |8 |
- +-----+--------------+-----+
- |B |BAM_CBACK |9 |
- +-----+--------------+-----+
+ +-----+----------------+-----+
+ |M |pysam.CMATCH |0 |
+ +-----+----------------+-----+
+ |I |pysam.CINS |1 |
+ +-----+----------------+-----+
+ |D |pysam.CDEL |2 |
+ +-----+----------------+-----+
+ |N |pysam.CREF_SKIP |3 |
+ +-----+----------------+-----+
+ |S |pysam.CSOFT_CLIP|4 |
+ +-----+----------------+-----+
+ |H |pysam.CHARD_CLIP|5 |
+ +-----+----------------+-----+
+ |P |pysam.CPAD |6 |
+ +-----+----------------+-----+
+ |= |pysam.CEQUAL |7 |
+ +-----+----------------+-----+
+ |X |pysam.CDIFF |8 |
+ +-----+----------------+-----+
+ |B |pysam.CBACK |9 |
+ +-----+----------------+-----+
.. note::
The output is a list of (operation, length) tuples, such as
from typing import Literal
from pysam.libchtslib import HTSFile, _HasFileNo
-from pysam.libcalignedsegment import AlignedSegment
+from pysam.libcalignedsegment import AlignedSegment, PileupColumn
from pysam.libcfaidx import FastaFile
class IndexStats(NamedTuple):
def lengths(self) -> Tuple[int]: ...
def to_dict(self) -> Dict: ...
def get_reference_name(self, tid: int) -> Optional[str]: ...
- def get_reference_length(self, reference: int) -> int: ...
+ def get_reference_length(self, reference: str) -> int: ...
def is_valid_tid(self, tid: int) -> bool: ...
def get_tid(self, reference: str) -> int: ...
class IteratorRowSelection(IteratorRow): ...
class IteratorColumn:
- def __iter__(self) -> IteratorRow: ...
+ def __iter__(self) -> IteratorColumn: ...
def __next__(self) -> PileupColumn: ...
@property
def seq_len(self) -> int: ...
stop: int = ...,
alleles: Optional[Tuple[str, ...]] = ...,
id: Optional[str] = ...,
- qual: Optional[int] = ...,
+ qual: Optional[float] = ...,
filter: Optional[Any] = ...,
info: Optional[Mapping[str, _InfoValue]] = ...,
samples: Optional[Iterable[Optional[Mapping[str, _FormatValue]]]] = ...,
start: int
stop: int
rlen: int
- qual: Optional[int]
+ qual: Optional[float]
id: Optional[str]
ref: Optional[str]
alleles: Optional[Tuple[str, ...]]
def index(self) -> int: ...
@property
def name(self) -> str: ...
- allele_indices: Optional[Tuple[Optional[int, ...]]]
- alleles: Optional[Tuple[Optional[str, ...]]]
+ allele_indices: Optional[Tuple[Optional[int], ...]]
+ alleles: Optional[Tuple[Optional[str], ...]]
phased: bool
def __setitem__(self, key: str, value: _FormatValue) -> None: ...
def __delitem__(self, key: str) -> None: ...
rec.info.update(info)
if kwargs:
- if 'GT' in kwargs:
- rec.samples[0]['GT'] = kwargs.pop('GT')
rec.samples[0].update(kwargs)
if samples:
for i, sample in enumerate(samples):
- if 'GT' in sample:
- rec.samples[i]['GT'] = sample.pop('GT')
rec.samples[i].update(sample)
return rec
--- /dev/null
+# cython: language_level=3
+
+from pysam.libchtslib cimport BGZF
+
+cdef class BGZFile(object):
+ cdef BGZF *bgzf
+ cdef readonly object name, index
This class only supports opening files in binary mode. If you need to open a
compressed file in text mode, use the gzip.open() function.
"""
- cdef BGZF* bgzf
- cdef readonly object name, index
-
def __init__(self, filename, mode=None, index=None):
"""Constructor for the BGZFile class.
# @param mode Open mode, as per hts_open()
htsFile *hts_hopen(hFILE *fp, const char *fn, const char *mode)
+ # @abstract For output streams, flush any buffered data
+ # @param fp The file handle to be flushed
+ # @return 0 for success, or negative if an error occurred.
+ # @since 1.14
+ int hts_flush(htsFile *fp)
+
# @abstract Close a file handle, flushing buffered data for output streams
# @param fp The file handle to be closed
# @return 0 for success, or negative if an error occurred.
@property
def duplicate_filehandle(self) -> bool: ...
def close(self) -> None: ...
+ def flush(self) -> None: ...
def check_truncation(self, ignore_truncation: bool = ...) -> None: ...
@property
def category(self) -> str: ...
hts_close(self.htsfile)
self.htsfile = NULL
+ def flush(self):
+ """Flush any buffered data to the underlying output stream."""
+ if self.htsfile:
+ if hts_flush(self.htsfile) < 0:
+ raise OSError(errno, f'Flushing {type(self).__name__} failed', force_str(self.filename))
+
def check_truncation(self, ignore_truncation=False):
"""Check if file is truncated."""
if not self.htsfile:
-from typing import Optional, Literal, List, Any
+import sys
+from typing import Optional, List, Any
+
+if sys.version_info < (3, 8):
+ from typing_extensions import Literal
+else:
+ from typing import Literal
from pysam.libchtslib import HTSFile
from cpython.version cimport PY_MAJOR_VERSION, PY_MINOR_VERSION
from cpython cimport PyBytes_Check, PyUnicode_Check
from cpython cimport array as c_array
+from libc.errno cimport errno
from libc.stdlib cimport calloc, free
from libc.string cimport strncpy
from libc.stdint cimport INT32_MAX, int32_t
from libc.stdio cimport fprintf, stderr, fflush
from libc.stdio cimport stdout as c_stdout
-from posix.fcntl cimport open as c_open, O_WRONLY
+from posix.fcntl cimport open as c_open, O_WRONLY, O_CREAT, O_TRUNC
from posix.unistd cimport SEEK_SET, SEEK_CUR, SEEK_END
from pysam.libcsamtools cimport samtools_dispatch, samtools_set_stdout, samtools_set_stderr, \
if save_stdout:
stdout_f = save_stdout
stdout_h = c_open(force_bytes(stdout_f),
- O_WRONLY)
+ O_WRONLY|O_CREAT|O_TRUNC, 0666)
if stdout_h == -1:
- raise IOError("error while opening {} for writing".format(stdout_f))
+ raise OSError(errno, "error while opening file for writing", stdout_f)
samtools_set_stdout_fn(force_bytes(stdout_f))
bcftools_set_stdout_fn(force_bytes(stdout_f))
if collection == "bcftools":
# in bcftools, most methods accept -o, the exceptions
# are below:
- if method not in ("index", "roh", "stats"):
+ if method not in ("head", "index", "roh", "stats"):
stdout_option = "-o {}"
elif method in MAP_STDOUT_OPTIONS[collection]:
# special case - samtools view -c outputs on stdout
-import platform
-from typing import (
- Callable,
- List,
- Tuple,
- Iterable,
- Union,
-)
-try:
- from typing import Final
- HAVE_FINAL = True
-except ImportError:
- HAVE_FINAL = False
-
-from pysam.utils import PysamDispatcher
-
-
-# samtools command line options to export in python
-_SAMTOOLS_DISPATCH = {
- # samtools 'documented' commands
- "view": ("view", ()),
- "head": ("head", ()),
- "sort": ("sort", ()),
- "mpileup": ("mpileup", ()),
- "consensus": ("consensus", ()),
- "depth": ("depth", ()),
- "faidx": ("faidx", ()),
- "fqidx": ("fqidx", ()),
- "tview": ("tview", ()),
- "index": ("index", ()),
- "idxstats": ("idxstats", ()),
- "fixmate": ("fixmate", ()),
- "flagstat": ("flagstat", ()),
- "calmd": ("calmd", ()),
- "merge": ("merge", ()),
- "markdup": ("markdup", ()),
- "rmdup": ("rmdup", ()),
- "reference": ("reference", ()),
- "reheader": ("reheader", ()),
- "reset": ("reset", ()),
- "cat": ("cat", ()),
- "targetcut": ("targetcut", ()),
- "phase": ("phase", ()),
- "bam2fq": ("bam2fq", ()),
- "dict": ("dict", ()),
- "addreplacerg": ("addreplacerg", ()),
- "pad2unpad": ("pad2unpad", ()),
- "depad": ("pad2unpad", ()),
- "bedcov": ("bedcov", ()),
- "coverage": ("coverage", ()),
- "bamshuf": ("bamshuf", ()),
- "collate": ("collate", ()),
- "stats": ("stats", ()),
- "fasta": ("fasta", ()),
- "fastq": ("fastq", ()),
- "cram_size": ("cram-size", ()),
- "quickcheck": ("quickcheck", ()),
- "split": ("split", ()),
- "flags": ("flags", ()),
- "ampliconclip": ("ampliconclip", ()),
- "ampliconstats": ("ampliconstats", ()),
- "version": ("version", ()),
- "fqimport": ("import", ()),
- "import_": ("import", ()),
- "samples": ("samples", ()),
-}
-
-
-def _wrap_command(
- dispatch: str,
- parsers: Iterable[Tuple[str, Callable[[Union[str, List[str]]], Union[str, List[str]]]]],
-) -> PysamDispatcher:
- return PysamDispatcher("samtools", dispatch, parsers)
-
-
-if not HAVE_FINAL:
- # python 3.7
- for key, options in _SAMTOOLS_DISPATCH.items():
- cmd, parser = options
- globals()[key] = PysamDispatcher("samtools", cmd, parser)
-
- __all__ = list(_SAMTOOLS_DISPATCH)
-else:
- # python >=3.8
- view: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["view"][0], _SAMTOOLS_DISPATCH["view"][1])
-
- head: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["head"][0], _SAMTOOLS_DISPATCH["head"][1])
-
- sort: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["sort"][0], _SAMTOOLS_DISPATCH["sort"][1])
-
- mpileup: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["mpileup"][0], _SAMTOOLS_DISPATCH["mpileup"][1])
-
- consensus: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["consensus"][0],
- _SAMTOOLS_DISPATCH["consensus"][1],
- )
-
- depth: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["depth"][0], _SAMTOOLS_DISPATCH["depth"][1])
-
- faidx: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["faidx"][0], _SAMTOOLS_DISPATCH["faidx"][1])
-
- fqidx: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["fqidx"][0], _SAMTOOLS_DISPATCH["fqidx"][1])
-
- tview: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["tview"][0], _SAMTOOLS_DISPATCH["tview"][1])
-
- index: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["index"][0], _SAMTOOLS_DISPATCH["index"][1])
-
- idxstats: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["idxstats"][0], _SAMTOOLS_DISPATCH["idxstats"][1])
-
- fixmate: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["fixmate"][0], _SAMTOOLS_DISPATCH["fixmate"][1])
-
- flagstat: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["flagstat"][0], _SAMTOOLS_DISPATCH["flagstat"][1])
-
- calmd: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["calmd"][0], _SAMTOOLS_DISPATCH["calmd"][1])
-
- merge: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["merge"][0], _SAMTOOLS_DISPATCH["merge"][1])
-
- markdup: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["markdup"][0], _SAMTOOLS_DISPATCH["markdup"][1])
-
- rmdup: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["rmdup"][0], _SAMTOOLS_DISPATCH["rmdup"][1])
-
- reference: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["reference"][0],
- _SAMTOOLS_DISPATCH["reference"][1],
- )
-
- reheader: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["reheader"][0], _SAMTOOLS_DISPATCH["reheader"][1])
-
- reset: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["reset"][0], _SAMTOOLS_DISPATCH["reset"][1])
-
- cat: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["cat"][0], _SAMTOOLS_DISPATCH["cat"][1])
-
- targetcut: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["targetcut"][0],
- _SAMTOOLS_DISPATCH["targetcut"][1],
- )
-
- phase: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["phase"][0], _SAMTOOLS_DISPATCH["phase"][1])
-
- bam2fq: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["bam2fq"][0], _SAMTOOLS_DISPATCH["bam2fq"][1])
-
- dict: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["dict"][0], _SAMTOOLS_DISPATCH["dict"][1])
-
- addreplacerg: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["addreplacerg"][0],
- _SAMTOOLS_DISPATCH["addreplacerg"][1],
- )
-
- pad2unpad: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["pad2unpad"][0],
- _SAMTOOLS_DISPATCH["pad2unpad"][1],
- )
-
- depad: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["depad"][0], _SAMTOOLS_DISPATCH["depad"][1])
-
- bedcov: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["bedcov"][0], _SAMTOOLS_DISPATCH["bedcov"][1])
-
- coverage: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["coverage"][0], _SAMTOOLS_DISPATCH["coverage"][1])
-
- bamshuf: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["bamshuf"][0], _SAMTOOLS_DISPATCH["bamshuf"][1])
-
- collate: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["collate"][0], _SAMTOOLS_DISPATCH["collate"][1])
-
- stats: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["stats"][0], _SAMTOOLS_DISPATCH["stats"][1])
-
- fasta: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["fasta"][0], _SAMTOOLS_DISPATCH["fasta"][1])
-
- fastq: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["fastq"][0], _SAMTOOLS_DISPATCH["fastq"][1])
-
- cram_size: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["cram_size"][0], _SAMTOOLS_DISPATCH["cram_size"][1])
-
- quickcheck: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["quickcheck"][0],
- _SAMTOOLS_DISPATCH["quickcheck"][1],
- )
-
- split: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["split"][0], _SAMTOOLS_DISPATCH["split"][1])
-
- flags: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["flags"][0], _SAMTOOLS_DISPATCH["flags"][1])
-
- ampliconclip: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["ampliconclip"][0],
- _SAMTOOLS_DISPATCH["ampliconclip"][1],
- )
-
- ampliconstats: Final[PysamDispatcher] = _wrap_command(
- _SAMTOOLS_DISPATCH["ampliconstats"][0],
- _SAMTOOLS_DISPATCH["ampliconstats"][1],
- )
-
- version: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["version"][0], _SAMTOOLS_DISPATCH["version"][1])
-
- fqimport: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["fqimport"][0], _SAMTOOLS_DISPATCH["fqimport"][1])
-
- import_: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["import_"][0], _SAMTOOLS_DISPATCH["import_"][1])
-
- samples: Final[PysamDispatcher] = _wrap_command(_SAMTOOLS_DISPATCH["samples"][0], _SAMTOOLS_DISPATCH["samples"][1])
+import pysam.utils
+
+addreplacerg = pysam.utils.PysamDispatcher('samtools', 'addreplacerg')
+ampliconclip = pysam.utils.PysamDispatcher('samtools', 'ampliconclip')
+ampliconstats = pysam.utils.PysamDispatcher('samtools', 'ampliconstats')
+bam2fq = pysam.utils.PysamDispatcher('samtools', 'bam2fq')
+bamshuf = pysam.utils.PysamDispatcher('samtools', 'bamshuf')
+bedcov = pysam.utils.PysamDispatcher('samtools', 'bedcov')
+calmd = pysam.utils.PysamDispatcher('samtools', 'calmd')
+cat = pysam.utils.PysamDispatcher('samtools', 'cat')
+collate = pysam.utils.PysamDispatcher('samtools', 'collate')
+consensus = pysam.utils.PysamDispatcher('samtools', 'consensus')
+coverage = pysam.utils.PysamDispatcher('samtools', 'coverage')
+cram_size = pysam.utils.PysamDispatcher('samtools', 'cram-size')
+depad = pysam.utils.PysamDispatcher('samtools', 'depad')
+depth = pysam.utils.PysamDispatcher('samtools', 'depth')
+dict = pysam.utils.PysamDispatcher('samtools', 'dict')
+faidx = pysam.utils.PysamDispatcher('samtools', 'faidx')
+fasta = pysam.utils.PysamDispatcher('samtools', 'fasta')
+fastq = pysam.utils.PysamDispatcher('samtools', 'fastq')
+fixmate = pysam.utils.PysamDispatcher('samtools', 'fixmate')
+flags = pysam.utils.PysamDispatcher('samtools', 'flags')
+flagstat = pysam.utils.PysamDispatcher('samtools', 'flagstat')
+fqidx = pysam.utils.PysamDispatcher('samtools', 'fqidx')
+fqimport = pysam.utils.PysamDispatcher('samtools', 'import')
+head = pysam.utils.PysamDispatcher('samtools', 'head')
+idxstats = pysam.utils.PysamDispatcher('samtools', 'idxstats')
+index = pysam.utils.PysamDispatcher('samtools', 'index')
+markdup = pysam.utils.PysamDispatcher('samtools', 'markdup')
+merge = pysam.utils.PysamDispatcher('samtools', 'merge')
+mpileup = pysam.utils.PysamDispatcher('samtools', 'mpileup')
+pad2unpad = pysam.utils.PysamDispatcher('samtools', 'pad2unpad')
+phase = pysam.utils.PysamDispatcher('samtools', 'phase')
+quickcheck = pysam.utils.PysamDispatcher('samtools', 'quickcheck')
+reference = pysam.utils.PysamDispatcher('samtools', 'reference')
+reheader = pysam.utils.PysamDispatcher('samtools', 'reheader')
+reset = pysam.utils.PysamDispatcher('samtools', 'reset')
+rmdup = pysam.utils.PysamDispatcher('samtools', 'rmdup')
+samples = pysam.utils.PysamDispatcher('samtools', 'samples')
+sort = pysam.utils.PysamDispatcher('samtools', 'sort')
+split = pysam.utils.PysamDispatcher('samtools', 'split')
+stats = pysam.utils.PysamDispatcher('samtools', 'stats')
+targetcut = pysam.utils.PysamDispatcher('samtools', 'targetcut')
+tview = pysam.utils.PysamDispatcher('samtools', 'tview')
+version = pysam.utils.PysamDispatcher('samtools', 'version')
+view = pysam.utils.PysamDispatcher('samtools', 'view')
+
+__all__ = [
+ 'addreplacerg', 'ampliconclip', 'ampliconstats',
+ 'bam2fq', 'bamshuf', 'bedcov', 'calmd', 'cat',
+ 'collate', 'consensus', 'coverage', 'cram_size',
+ 'depad', 'depth', 'dict', 'faidx', 'fasta',
+ 'fastq', 'fixmate', 'flags', 'flagstat', 'fqidx',
+ 'fqimport', 'head', 'idxstats', 'index',
+ 'markdup', 'merge', 'mpileup', 'pad2unpad',
+ 'phase', 'quickcheck', 'reference', 'reheader',
+ 'reset', 'rmdup', 'samples', 'sort', 'split',
+ 'stats', 'targetcut', 'tview', 'version', 'view',
+]
from typing import (
Callable,
List,
+ Optional,
Tuple,
Iterable,
Union,
self,
collection: str,
dispatch: str,
- parsers: Iterable[Tuple[str, Callable[[Union[str, List[str]]], Union[str, List[str]]]]],
+ parsers: Optional[Iterable[Tuple[str, Callable[[Union[str, List[str]]], Union[str, List[str]]]]]] = None,
):
self.collection = collection
self.dispatch = dispatch
// Version information used while compiling samtools, bcftools, and htslib
-#define SAMTOOLS_VERSION "1.18 (pysam)"
-#define BCFTOOLS_VERSION "1.18 (pysam)"
-#define HTS_VERSION_TEXT "1.18 (pysam)"
+#define SAMTOOLS_VERSION "1.21 (pysam)"
+#define BCFTOOLS_VERSION "1.21 (pysam)"
+#define HTS_VERSION_TEXT "1.21 (pysam)"
# pysam versioning information
-__version__ = "0.22.1"
+__version__ = "0.23.0"
-__samtools_version__ = "1.18"
-__bcftools_version__ = "1.18"
-__htslib_version__ = "1.18"
+__samtools_version__ = "1.21"
+__bcftools_version__ = "1.21"
+__htslib_version__ = "1.21"
The MIT/Expat License
-Copyright (C) 2008-2023 Genome Research Ltd.
+Copyright (C) 2008-2024 Genome Research Ltd.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
The typical simple case of building Samtools using the HTSlib bundled within
this Samtools release tarball is done as follows:
- cd .../samtools-1.18 # Within the unpacked release directory
+ cd .../samtools-1.21 # Within the unpacked release directory
./configure
make
installation using the HTSlib bundled within this Samtools release tarball,
and building the various HTSlib utilities such as bgzip is done as follows:
- cd .../samtools-1.18 # Within the unpacked release directory
+ cd .../samtools-1.21 # Within the unpacked release directory
./configure --prefix=/path/to/location
make all all-htslib
make install install-htslib
To build with plug-ins, you need to use the --enable-plugins configure option
as follows:
- cd .../samtools-1.18 # Within the unpacked release directory
+ cd .../samtools-1.21 # Within the unpacked release directory
./configure --enable-plugins --prefix=/path/to/location
make all all-htslib
make install install-htslib
the source distribution instead of installing the package. In that case
you can use:
- cd .../samtools-1.18 # Within the unpacked release directory
- ./configure --enable-plugins --with-plugin-path=$PWD/htslib-1.18
+ cd .../samtools-1.21 # Within the unpacked release directory
+ ./configure --enable-plugins --with-plugin-path=$PWD/htslib-1.21
make all all-htslib
It is possible to override the built-in search path using the HTS_PATH
/* stats.c -- This is the former bamcheck integrated into samtools/htslib.
- Copyright (C) 2020-2021 Genome Research Ltd.
+ Copyright (C) 2020-2021, 2024 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
{"single-ref", no_argument, NULL, 'S'},
{NULL, 0, NULL, 0}
};
- int opt;
+ int opt, tmp_flag;
while ( (opt=getopt_long(argc,argv,"?hf:F:@:p:m:d:sa:l:t:o:c:b:D:S",loptions,NULL))>0 ) {
switch (opt) {
- case 'f': args.flag_require = bam_str2flag(optarg); break;
+ case 'f':
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("ampliconstats", "Unknown flag '%s'\n", optarg);
+ return 1;
+ }
+
+ args.flag_require = tmp_flag;
+ break;
+
case 'F':
+ tmp_flag = bam_str2flag(optarg);
+
if (args.flag_filter & 0x10000)
args.flag_filter = 0; // strip default on first -F usage
- args.flag_filter |= bam_str2flag(optarg); break;
+
+ if (tmp_flag < 0) {
+ print_error("ampliconstats", "Unknown flag '%s'\n", optarg);
+ return 1;
+ }
+
+ args.flag_filter |= tmp_flag;
+ break;
case 'm': args.max_delta = atoi(optarg); break; // margin
case 'D': args.depth_bin = atof(optarg); break; // depth bin fraction
return usage(&oargs, stderr, EXIT_FAILURE);
khash_t(bed_list_hash) *bed_hash = kh_init(bed_list_hash);
- if (load_bed_file_multi_ref(argv[optind], 1, 0, bed_hash)) {
+ if (load_bed_file_multi_ref(argv[optind], 1, 0, bed_hash, NULL, NULL)) {
print_error_errno("ampliconstats",
"Could not read file \"%s\"", argv[optind]);
return 1;
/* stats.c -- This is the former bamcheck integrated into samtools/htslib.
- Copyright (C) 2020-2021 Genome Research Ltd.
+ Copyright (C) 2020-2021, 2024 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
{"single-ref", no_argument, NULL, 'S'},
{NULL, 0, NULL, 0}
};
- int opt;
+ int opt, tmp_flag;
while ( (opt=getopt_long(argc,argv,"?hf:F:@:p:m:d:sa:l:t:o:c:b:D:S",loptions,NULL))>0 ) {
switch (opt) {
- case 'f': args.flag_require = bam_str2flag(optarg); break;
+ case 'f':
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("ampliconstats", "Unknown flag '%s'\n", optarg);
+ return 1;
+ }
+
+ args.flag_require = tmp_flag;
+ break;
+
case 'F':
+ tmp_flag = bam_str2flag(optarg);
+
if (args.flag_filter & 0x10000)
args.flag_filter = 0; // strip default on first -F usage
- args.flag_filter |= bam_str2flag(optarg); break;
+
+ if (tmp_flag < 0) {
+ print_error("ampliconstats", "Unknown flag '%s'\n", optarg);
+ return 1;
+ }
+
+ args.flag_filter |= tmp_flag;
+ break;
case 'm': args.max_delta = atoi(optarg); break; // margin
case 'D': args.depth_bin = atof(optarg); break; // depth bin fraction
return usage(&oargs, samtools_stderr, EXIT_FAILURE);
khash_t(bed_list_hash) *bed_hash = kh_init(bed_list_hash);
- if (load_bed_file_multi_ref(argv[optind], 1, 0, bed_hash)) {
+ if (load_bed_file_multi_ref(argv[optind], 1, 0, bed_hash, NULL, NULL)) {
print_error_errno("ampliconstats",
"Could not read file \"%s\"", argv[optind]);
return 1;
/* bam2depth.c -- depth subcommand.
Copyright (C) 2011, 2012 Broad Institute.
- Copyright (C) 2012-2016, 2018, 2019-2022 Genome Research Ltd.
+ Copyright (C) 2012-2016, 2018, 2019-2022, 2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk> (to 2020)
Author: James Bonfield <jkb@sanger.ac.uk> (2021 rewrite)
}
+// Without HTS_OPT3 gcc11 and earlier don't vectorize this, nor even
+// unroll it. By adding HTS_OPT3 we can force a better level of optimization.
+// On an Illumina BAM with gcc11 -O2, with HTS_OPT3 is 9% quicker for the
+// entire process (ie decompress, iterator, aggregate & report)
+static inline void HTS_OPT3 incr_hist(int *hist, int oplen) {
+ const int N = 16;
+ int k;
+ for (k = 0; k < (oplen & ~(N-1)); k+=N) {
+ for (int i = 0; i < N; i++)
+ hist[k+i]++;
+ }
+ for (; k < oplen; k++)
+ hist[k]++;
+}
+
+static inline void HTS_OPT3
+incr_hist_qual(int *hist, uint8_t *qual, int min_qual, int oplen) {
+ if (!min_qual) {
+ incr_hist(hist, oplen);
+ return;
+ }
+
+ int k;
+ for (k = 0; k < (oplen & ~31); k+=32) {
+ int pass[32];
+ // Two separate loops helps clang to vectorize
+ for (int i = 0; i < 32; i++)
+ pass[i]=qual[k+i]>=min_qual;
+
+ for (int i = 0; i < 32; i++)
+ hist[k+i]+=pass[i];
+ }
+ for (; k < oplen; k++)
+ hist[k]+=qual[k]>=min_qual;
+}
+
// Adds the depth for a single read to a depth_hist struct.
// For just one file, this is easy. We just have a circular buffer
// where we increment values for bits that overlap existing data
// We've explicitly asked to include them, and the quality
// is wrong anyway (it's the neighbouring base). We do this
// for now for compatibility with the old depth command.
-
if (spos < b->core.l_qseq)
for (; k < oplen; k++, i++)
hist[i & hmask]+=qual[spos]>=min_qual;
case BAM_CMATCH:
case BAM_CEQUAL:
- case BAM_CDIFF:
- if ((i & hmask) < ((i+oplen) & hmask)) {
- // Optimisation when not wrapping around
-
- // Unrolling doesn't help clang, but helps gcc,
- // especially when not using -O3.
- int *hist = &dh->hist[file][i & hmask];
- if (min_qual || overlap_clip) {
- k = 0;
- if (overlap_clip) {
- if (i+oplen < overlap_clip) {
- i += oplen;
- spos += oplen;
- break;
- } else if (i < overlap_clip) {
- oplen -= overlap_clip - i;
- spos += overlap_clip - i;
- hist += overlap_clip - i;
- i = overlap_clip;
- }
- }
-
- // approx 50% of this func cpu time in this loop
- for (; k < (oplen & ~7); k+=8) {
- hist[k+0]+=qual[spos+0]>=min_qual;
- hist[k+1]+=qual[spos+1]>=min_qual;
- hist[k+2]+=qual[spos+2]>=min_qual;
- hist[k+3]+=qual[spos+3]>=min_qual;
- hist[k+4]+=qual[spos+4]>=min_qual;
- hist[k+5]+=qual[spos+5]>=min_qual;
- hist[k+6]+=qual[spos+6]>=min_qual;
- hist[k+7]+=qual[spos+7]>=min_qual;
- spos += 8;
- }
- } else {
- // easier to vectorize when no min_qual
- for (k = 0; k < (oplen & ~7); k+=8) {
- hist[k+0]++;
- hist[k+1]++;
- hist[k+2]++;
- hist[k+3]++;
- hist[k+4]++;
- hist[k+5]++;
- hist[k+6]++;
- hist[k+7]++;
- }
- spos += k;
- }
- for (; k < oplen && spos < b->core.l_qseq; k++, spos++)
- hist[k]+=qual[spos]>=min_qual;
- for (; k < oplen; k++, spos++)
- hist[k]++;
- i += oplen;
- } else {
- // Simple to understand case, but slower.
- // We use this only for reads with wrap-around.
- int *hist = dh->hist[file];
- k = 0;
- if (overlap_clip) {
- if (i+oplen < overlap_clip) {
- i += oplen;
- break;
- } else if (i < overlap_clip) {
- oplen -= overlap_clip - i;
- spos += overlap_clip - i;
- i = overlap_clip;
- }
+ case BAM_CDIFF: {
+ int *hist = dh->hist[file];
+ k = 0;
+ if (overlap_clip) {
+ if (i+oplen < overlap_clip) {
+ i += oplen;
+ break;
+ } else if (i < overlap_clip) {
+ oplen -= overlap_clip - i;
+ spos += overlap_clip - i;
+ i = overlap_clip;
}
- for (; k < oplen && spos < b->core.l_qseq; k++, i++, spos++)
- hist[i & hmask]+=qual[spos]>=min_qual;
- for (; k < oplen; k++, i++, spos++)
- hist[i & hmask]++;
}
+
+ int len = (i & hmask) < ((i+oplen) & hmask)
+ ? oplen // doesn't wrap around
+ : dh->size - (i&hmask); // does wrap around
+ incr_hist_qual(&hist[i & hmask], &qual[spos], min_qual, len);
+ if (oplen > len)
+ incr_hist_qual(hist, &qual[spos+len], min_qual, oplen-len);
+ spos += oplen;
+ i += oplen;
break;
+ }
case BAM_CINS:
case BAM_CSOFT_CLIP:
int main_depth(int argc, char *argv[])
{
- int nfiles, i;
+ int nfiles, i, tmp_flag;
samFile **fp;
sam_hdr_t **header;
int c, has_index_file = 0;
break;
case 'g':
- opt.flag &= ~bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.flag &= ~tmp_flag;
break;
case 'G': // reject if any set
- opt.flag |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.flag |= tmp_flag;
break;
case 1: // reject unless at least one set (0 means ignore option)
- opt.incl_flag |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.incl_flag |= tmp_flag;
break;
case 2: // reject unless all set
- opt.require_flag |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.require_flag |= tmp_flag;
break;
case 'l':
/* bam2depth.c -- depth subcommand.
Copyright (C) 2011, 2012 Broad Institute.
- Copyright (C) 2012-2016, 2018, 2019-2022 Genome Research Ltd.
+ Copyright (C) 2012-2016, 2018, 2019-2022, 2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk> (to 2020)
Author: James Bonfield <jkb@sanger.ac.uk> (2021 rewrite)
}
+// Without HTS_OPT3 gcc11 and earlier don't vectorize this, nor even
+// unroll it. By adding HTS_OPT3 we can force a better level of optimization.
+// On an Illumina BAM with gcc11 -O2, with HTS_OPT3 is 9% quicker for the
+// entire process (ie decompress, iterator, aggregate & report)
+static inline void HTS_OPT3 incr_hist(int *hist, int oplen) {
+ const int N = 16;
+ int k;
+ for (k = 0; k < (oplen & ~(N-1)); k+=N) {
+ for (int i = 0; i < N; i++)
+ hist[k+i]++;
+ }
+ for (; k < oplen; k++)
+ hist[k]++;
+}
+
+static inline void HTS_OPT3
+incr_hist_qual(int *hist, uint8_t *qual, int min_qual, int oplen) {
+ if (!min_qual) {
+ incr_hist(hist, oplen);
+ return;
+ }
+
+ int k;
+ for (k = 0; k < (oplen & ~31); k+=32) {
+ int pass[32];
+ // Two separate loops helps clang to vectorize
+ for (int i = 0; i < 32; i++)
+ pass[i]=qual[k+i]>=min_qual;
+
+ for (int i = 0; i < 32; i++)
+ hist[k+i]+=pass[i];
+ }
+ for (; k < oplen; k++)
+ hist[k]+=qual[k]>=min_qual;
+}
+
// Adds the depth for a single read to a depth_hist struct.
// For just one file, this is easy. We just have a circular buffer
// where we increment values for bits that overlap existing data
// We've explicitly asked to include them, and the quality
// is wrong anyway (it's the neighbouring base). We do this
// for now for compatibility with the old depth command.
-
if (spos < b->core.l_qseq)
for (; k < oplen; k++, i++)
hist[i & hmask]+=qual[spos]>=min_qual;
case BAM_CMATCH:
case BAM_CEQUAL:
- case BAM_CDIFF:
- if ((i & hmask) < ((i+oplen) & hmask)) {
- // Optimisation when not wrapping around
-
- // Unrolling doesn't help clang, but helps gcc,
- // especially when not using -O3.
- int *hist = &dh->hist[file][i & hmask];
- if (min_qual || overlap_clip) {
- k = 0;
- if (overlap_clip) {
- if (i+oplen < overlap_clip) {
- i += oplen;
- spos += oplen;
- break;
- } else if (i < overlap_clip) {
- oplen -= overlap_clip - i;
- spos += overlap_clip - i;
- hist += overlap_clip - i;
- i = overlap_clip;
- }
- }
-
- // approx 50% of this func cpu time in this loop
- for (; k < (oplen & ~7); k+=8) {
- hist[k+0]+=qual[spos+0]>=min_qual;
- hist[k+1]+=qual[spos+1]>=min_qual;
- hist[k+2]+=qual[spos+2]>=min_qual;
- hist[k+3]+=qual[spos+3]>=min_qual;
- hist[k+4]+=qual[spos+4]>=min_qual;
- hist[k+5]+=qual[spos+5]>=min_qual;
- hist[k+6]+=qual[spos+6]>=min_qual;
- hist[k+7]+=qual[spos+7]>=min_qual;
- spos += 8;
- }
- } else {
- // easier to vectorize when no min_qual
- for (k = 0; k < (oplen & ~7); k+=8) {
- hist[k+0]++;
- hist[k+1]++;
- hist[k+2]++;
- hist[k+3]++;
- hist[k+4]++;
- hist[k+5]++;
- hist[k+6]++;
- hist[k+7]++;
- }
- spos += k;
- }
- for (; k < oplen && spos < b->core.l_qseq; k++, spos++)
- hist[k]+=qual[spos]>=min_qual;
- for (; k < oplen; k++, spos++)
- hist[k]++;
- i += oplen;
- } else {
- // Simple to understand case, but slower.
- // We use this only for reads with wrap-around.
- int *hist = dh->hist[file];
- k = 0;
- if (overlap_clip) {
- if (i+oplen < overlap_clip) {
- i += oplen;
- break;
- } else if (i < overlap_clip) {
- oplen -= overlap_clip - i;
- spos += overlap_clip - i;
- i = overlap_clip;
- }
+ case BAM_CDIFF: {
+ int *hist = dh->hist[file];
+ k = 0;
+ if (overlap_clip) {
+ if (i+oplen < overlap_clip) {
+ i += oplen;
+ break;
+ } else if (i < overlap_clip) {
+ oplen -= overlap_clip - i;
+ spos += overlap_clip - i;
+ i = overlap_clip;
}
- for (; k < oplen && spos < b->core.l_qseq; k++, i++, spos++)
- hist[i & hmask]+=qual[spos]>=min_qual;
- for (; k < oplen; k++, i++, spos++)
- hist[i & hmask]++;
}
+
+ int len = (i & hmask) < ((i+oplen) & hmask)
+ ? oplen // doesn't wrap around
+ : dh->size - (i&hmask); // does wrap around
+ incr_hist_qual(&hist[i & hmask], &qual[spos], min_qual, len);
+ if (oplen > len)
+ incr_hist_qual(hist, &qual[spos+len], min_qual, oplen-len);
+ spos += oplen;
+ i += oplen;
break;
+ }
case BAM_CINS:
case BAM_CSOFT_CLIP:
int main_depth(int argc, char *argv[])
{
- int nfiles, i;
+ int nfiles, i, tmp_flag;
samFile **fp;
sam_hdr_t **header;
int c, has_index_file = 0;
break;
case 'g':
- opt.flag &= ~bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.flag &= ~tmp_flag;
break;
case 'G': // reject if any set
- opt.flag |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.flag |= tmp_flag;
break;
case 1: // reject unless at least one set (0 means ignore option)
- opt.incl_flag |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.incl_flag |= tmp_flag;
break;
case 2: // reject unless all set
- opt.require_flag |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("depth", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ opt.require_flag |= tmp_flag;
break;
case 'l':
/* bam_ampliconclip.c -- loads amplicon primers from a BED file and cuts reads
from the 5' end.
- Copyright (C) 2020-2023 Genome Research Ltd.
+ Copyright (C) 2020-2024 Genome Research Ltd.
Authors: Andrew Whitwham <aw7@sanger.ac.uk>
Rob Davies <rmd+git@sanger.ac.uk>
int unmap_len;
char *arg_list;
char *stats_file;
+ char *primer_counts_file;
char *rejects_file;
} cl_param_t;
return a->right < b->right ? -1 : (a->right == b->right ? 0 : 1);
}
-
-int load_bed_file_multi_ref(char *infile, int get_strand, int sort_by_pos, khash_t(bed_list_hash) *bed_lists) {
+int load_bed_file_multi_ref(char *infile, int get_strand, int sort_by_pos, khash_t(bed_list_hash) *bed_lists, char ***ref_list, size_t *num_refs_out) {
hFILE *fp;
int line_count = 0, ret;
+
+ //variables to store the bed file data for each record
+ char ref[1024] = "";
int64_t left, right;
+ char name[1024] = "", score[1024] = "";
+ char strand;
+
+ //hash table to store clipping results and bed file data
kstring_t line = KS_INITIALIZE;
bed_entry_list_t *list;
khiter_t bed_itr;
+ //ordered ref names list
+ size_t ref_list_sz = 0;
+ size_t num_refs = 0;
+
+ if (ref_list)
+ *ref_list = NULL;
+
+ if (num_refs_out)
+ *num_refs_out = 0;
+
if ((fp = hopen(infile, "r")) == NULL) {
print_error_errno("amplicon", "unable to open file %s.", infile);
return 1;
}
- char ref[1024];
while (line.l = 0, kgetline(&line, (kgets_func *)hgets, fp) >= 0) {
line_count++;
int hret;
- char strand;
if (line.l == 0 || *line.s == '#') continue;
if (strncmp(line.s, "track ", 6) == 0) continue;
if (strncmp(line.s, "browser ", 8) == 0) continue;
- if (get_strand) {
- if (sscanf(line.s, "%1023s %"SCNd64" %"SCNd64" %*s %*s %c",
- ref, &left, &right, &strand) != 4) {
- fprintf(stderr, "[amplicon] error: bad bed file format in line %d of %s.\n"
- "(N.B. ref/chrom name limited to 1023 characters.)\n",
- line_count, infile);
- ret = 1;
- goto error;
- }
- } else {
- if (sscanf(line.s, "%1023s %"SCNd64" %"SCNd64,
- ref, &left, &right) != 3) {
- fprintf(stderr, "[amplicon] error: bad bed file format in line %d of %s\n"
- "(N.B. ref/chrom name limited to 1023 characters.)\n",
- line_count, infile);
- ret = 1;
- goto error;
- }
+ // A list of the maximal number of columns we may want to parse.
+ // There may be more, but we don't use the ones beyond these.
+ const char *const scanf_str =
+ "%1023s %"SCNd64" %"SCNd64" %1023s %1023s %c";
+
+ // Extract the data from the line into the variables.
+ // Variables corresponding to any missing columns will remain
+ // uninitialised. We asked for all columns, but cols_parsed will
+ // return how many we found which can be validated against num_columns.
+ int cols_parsed = sscanf(line.s, scanf_str, ref,
+ &left, &right, name, score, &strand);
+ if (cols_parsed < (get_strand ? 6 : 3)) {
+ fprintf(stderr, "[amplicon] error: invalid bed file format in line %d of %s.\n"
+ "Parsed %d columns, but need at least %d\n"
+ "(N.B. ref/chrom name limited to 1023 characters.)\n",
+ line_count, infile, cols_parsed, get_strand ? 6 : 3);
+ ret = 1;
+ goto error;
}
bed_itr = kh_get(bed_list_hash, bed_lists, ref);
goto error;
}
+ if (ref_list) {
+ if (hts_resize(char **, num_refs + 1, &ref_list_sz, ref_list, 0) < 0) {
+ fprintf(stderr, "[amplicon] error: unable to allocate memory for ref name list.\n");
+ ret = 1;
+ goto error;
+ }
+ (*ref_list)[num_refs++] = ref_name;
+ }
bed_itr = kh_put(bed_list_hash, bed_lists, ref_name, &hret);
if (hret > 0) {
list = &kh_val(bed_lists, bed_itr);
}
+ // add the bed entry to the list, growing the list if necessary
if (list->length == list->size) {
bed_entry_t *tmp;
list->bp[list->length].left = left;
list->bp[list->length].right = right;
+ list->bp[list->length].name = NULL;
+ list->bp[list->length].score = NULL;
+ if (cols_parsed >= 4) {
+ list->bp[list->length].name = strdup(name);
+ if (list->bp[list->length].name == NULL) {
+ fprintf(stderr, "[amplicon] error: unable to allocate memory for name in line %d of %s: %s.\n", line_count, infile, line.s);
+ ret = 1;
+ goto error;
+ }
+ }
+ if (cols_parsed >= 5) {
+ list->bp[list->length].score = strdup(score);
+ if (list->bp[list->length].score == NULL) {
+ fprintf(stderr, "[amplicon] error: unable to allocate memory for score in line %d of %s: %s\n", line_count, infile, line.s);
+ ret = 1;
+ goto error;
+ }
+ }
+
+ list->bp[list->length].num_reads = 0;
if (get_strand) {
if (strand == '+') {
ret = 1;
}
+ if (num_refs_out)
+ *num_refs_out = num_refs;
+
error:
ks_free(&line);
for (itr = kh_begin(hash); itr != kh_end(hash); ++itr) {
if (kh_exist(hash, itr)) {
- free(kh_val(hash, itr).bp);
+ bed_entry_list_t list = kh_val(hash, itr);
+ for (int i = 0; i < list.length; i++) {
+ free(list.bp[i].name);
+ free(list.bp[i].score);
+ }
+ free(list.bp);
free((char *)kh_key(hash, itr));
kh_key(hash, itr) = NULL;
}
static int matching_clip_site(bed_entry_list_t *sites, hts_pos_t pos,
int is_rev, int use_strand, int64_t longest,
cl_param_t *param) {
- int i, size; // may need this to be variable
+ int i, size, used_i;
int tol = param->tol;
int l = 0, mid = sites->length / 2, r = sites->length;
int pos_tol = is_rev ? (pos > tol ? pos - tol : 0) : pos;
}
size = 0;
+ used_i = -1;
for (i = l; i < sites->length; i++) {
hts_pos_t mod_left, mod_right;
if (is_rev) {
if (size < pos - sites->bp[i].left) {
size = pos - sites->bp[i].left;
+ used_i = i;
}
} else {
if (size < sites->bp[i].right - pos) {
size = sites->bp[i].right - pos;
+ used_i = i;
}
}
}
}
-
+ if (used_i >= 0) {
+ sites->bp[used_i].num_reads++;
+ }
return size;
}
}
// Copy remaining QUAL
- memmove(new_qual, orig_qual, rec->core.l_qseq - qry_removed);
+ memmove(new_qual, orig_qual + qry_removed, rec->core.l_qseq - qry_removed);
// Set new l_qseq
rec_out->core.l_qseq -= qry_removed;
kstring_t seq = KS_INITIALIZE;
bed_entry_list_t *sites;
FILE *stats_fp = stderr;
+ FILE *bed_count_summary_fp = stderr;
khash_t(bed_list_hash) *bed_hash = kh_init(bed_list_hash);
+ char **bed_ref_list = NULL;
+ size_t num_bed_refs = 0;
- if (load_bed_file_multi_ref(bedfile, param->use_strand, 1, bed_hash)) {
+ if (load_bed_file_multi_ref(bedfile, param->use_strand, 1, bed_hash,
+ &bed_ref_list, &num_bed_refs)) {
fprintf(stderr, "[ampliconclip] error: unable to load bed file.\n");
goto fail;
}
if (file_open) {
fclose(stats_fp);
+ file_open = 0;
+ }
+
+ if (param->primer_counts_file) {
+ if ((bed_count_summary_fp = fopen(param->primer_counts_file, "w")) == NULL) {
+ fprintf(stderr, "[ampliconclip] warning: cannot write count summary to %s.\n", param->primer_counts_file);
+ } else {
+ file_open = 1;
+ }
+
+ //print out the number of reads for each bed entry, bedgraph format
+ fprintf(bed_count_summary_fp, "#CHR\tLEFT\tRIGHT\tNAME\tSCORE\tSTRAND\tNUM_CLIPPED\n");
+ size_t refidx;
+ for (refidx = 0; refidx < num_bed_refs; refidx++) {
+ khiter_t itr = kh_get(bed_list_hash, bed_hash, bed_ref_list[refidx]);
+ if (itr >= kh_end(bed_hash)) {
+ fprintf(stderr, "[ampliconclip] error: %s has gone missing from the hash table\n", bed_ref_list[refidx]);
+ goto fail;
+ }
+ sites = &kh_val(bed_hash, itr);
+ int i;
+ for (i = 0; i < sites->length; i++) {
+ char* strand_out = param->use_strand ? (sites->bp[i].rev ? "-" : "+") : ".";
+ fprintf(bed_count_summary_fp, "%s\t%"PRId64"\t%"PRId64"\t%s\t%s\t%s\t%"PRId64"\n",
+ kh_key(bed_hash, itr), sites->bp[i].left, sites->bp[i].right, sites->bp[i].name,
+ sites->bp[i].score, strand_out, sites->bp[i].num_reads);
+ }
+ }
+ if (file_open) {
+ fclose(bed_count_summary_fp);
+ }
}
ret = 0;
fail:
+ free(bed_ref_list);
destroy_bed_hash(bed_hash);
ks_free(&oat);
ks_free(&seq);
static void usage(void) {
fprintf(stderr, "Usage: samtools ampliconclip -b BED file <input.bam> -o <output.bam>\n\n");
fprintf(stderr, "Option: \n");
- fprintf(stderr, " -b FILE BED file of regions (eg amplicon primers) to be removed.\n");
- fprintf(stderr, " -o FILE output file name (default stdout).\n");
- fprintf(stderr, " -f FILE write stats to file name (default stderr)\n");
- fprintf(stderr, " -u Output uncompressed data\n");
- fprintf(stderr, " --soft-clip soft clip amplicon primers from reads (default)\n");
- fprintf(stderr, " --hard-clip hard clip amplicon primers from reads.\n");
- fprintf(stderr, " --both-ends clip on both 5' and 3' ends.\n");
- fprintf(stderr, " --strand use strand data from BED file to match read direction.\n");
- fprintf(stderr, " --clipped only output clipped reads.\n");
- fprintf(stderr, " --fail mark unclipped, mapped reads as QCFAIL.\n");
- fprintf(stderr, " --filter-len INT do not output reads INT size or shorter.\n");
- fprintf(stderr, " --fail-len INT mark as QCFAIL reads INT size or shorter.\n");
- fprintf(stderr, " --unmap-len INT unmap reads INT size or shorter, default 0.\n");
- fprintf(stderr, " --no-excluded do not write excluded reads (unmapped or QCFAIL).\n");
- fprintf(stderr, " --rejects-file FILE file to write filtered reads.\n");
- fprintf(stderr, " --original for clipped entries add an OA tag with original data.\n");
- fprintf(stderr, " --keep-tag for clipped entries keep the old NM and MD tags.\n");
- fprintf(stderr, " --tolerance match region within this number of bases, default 5.\n");
- fprintf(stderr, " --no-PG do not add an @PG line.\n");
+ fprintf(stderr, " -b FILE BED file of regions (eg amplicon primers) to be removed.\n");
+ fprintf(stderr, " -o FILE output file name (default: stdout).\n");
+ fprintf(stderr, " -f FILE write stats to file name (default: stderr)\n");
+ fprintf(stderr, " -u Output uncompressed data\n");
+ fprintf(stderr, " --soft-clip soft clip amplicon primers from reads (default)\n");
+ fprintf(stderr, " --hard-clip hard clip amplicon primers from reads.\n");
+ fprintf(stderr, " --both-ends clip on both 5' and 3' ends.\n");
+ fprintf(stderr, " --strand use strand data from BED file to match read direction.\n");
+ fprintf(stderr, " --clipped only output clipped reads.\n");
+ fprintf(stderr, " --fail mark unclipped, mapped reads as QCFAIL.\n");
+ fprintf(stderr, " --filter-len INT do not output reads INT size or shorter.\n");
+ fprintf(stderr, " --fail-len INT mark as QCFAIL reads INT size or shorter.\n");
+ fprintf(stderr, " --unmap-len INT unmap reads INT size or shorter, default 0.\n");
+ fprintf(stderr, " --no-excluded do not write excluded reads (unmapped or QCFAIL).\n");
+ fprintf(stderr, " --rejects-file FILE file to write filtered reads.\n");
+ fprintf(stderr, " --primer-counts FILE file to write read counts per bed entry (bedgraph format).\n");
+ fprintf(stderr, " --original for clipped entries add an OA tag with original data.\n");
+ fprintf(stderr, " --keep-tag for clipped entries keep the old NM and MD tags.\n");
+ fprintf(stderr, " --tolerance match region within this number of bases, default 5.\n");
+ fprintf(stderr, " --no-PG do not add an @PG line.\n");
sam_global_opt_help(stderr, "-.O..@-.");
fprintf(stderr, "\nAbout: Soft clips read alignments where they match BED file defined regions.\n"
"Default clipping is only on the 5' end.\n\n");
{"fail-len", required_argument, NULL, 1010},
{"no-excluded", no_argument, NULL, 1011},
{"rejects-file", required_argument, NULL, 1012},
- {"original", no_argument, NULL, 1013},
- {"keep-tag", no_argument, NULL, 1014},
- {"tolerance", required_argument, NULL, 1015},
- {"unmap-len", required_argument, NULL, 1016},
+ {"primer-counts", required_argument, NULL, 1013},
+ {"original", no_argument, NULL, 1014},
+ {"keep-tag", no_argument, NULL, 1015},
+ {"tolerance", required_argument, NULL, 1016},
+ {"unmap-len", required_argument, NULL, 1017},
{NULL, 0, NULL, 0}
};
case 1010: param.fail_len = atoi(optarg); break;
case 1011: param.unmapped = 1; break;
case 1012: param.rejects_file = optarg; break;
- case 1013: param.oa_tag = 1; break;
- case 1014: param.del_tag = 0; break;
- case 1015: param.tol = atoi(optarg); break;
- case 1016: param.unmap_len = atoi(optarg); break;
+ case 1013: param.primer_counts_file = optarg; break;
+ case 1014: param.oa_tag = 1; break;
+ case 1015: param.del_tag = 0; break;
+ case 1016: param.tol = atoi(optarg); break;
+ case 1017: param.unmap_len = atoi(optarg); break;
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
/* else fall-through */
case '?': usage(); exit(1);
/* bam_ampliconclip.c -- loads amplicon primers from a BED file and cuts reads
from the 5' end.
- Copyright (C) 2020-2023 Genome Research Ltd.
+ Copyright (C) 2020-2024 Genome Research Ltd.
Authors: Andrew Whitwham <aw7@sanger.ac.uk>
Rob Davies <rmd+git@sanger.ac.uk>
int unmap_len;
char *arg_list;
char *stats_file;
+ char *primer_counts_file;
char *rejects_file;
} cl_param_t;
return a->right < b->right ? -1 : (a->right == b->right ? 0 : 1);
}
-
-int load_bed_file_multi_ref(char *infile, int get_strand, int sort_by_pos, khash_t(bed_list_hash) *bed_lists) {
+int load_bed_file_multi_ref(char *infile, int get_strand, int sort_by_pos, khash_t(bed_list_hash) *bed_lists, char ***ref_list, size_t *num_refs_out) {
hFILE *fp;
int line_count = 0, ret;
+
+ //variables to store the bed file data for each record
+ char ref[1024] = "";
int64_t left, right;
+ char name[1024] = "", score[1024] = "";
+ char strand;
+
+ //hash table to store clipping results and bed file data
kstring_t line = KS_INITIALIZE;
bed_entry_list_t *list;
khiter_t bed_itr;
+ //ordered ref names list
+ size_t ref_list_sz = 0;
+ size_t num_refs = 0;
+
+ if (ref_list)
+ *ref_list = NULL;
+
+ if (num_refs_out)
+ *num_refs_out = 0;
+
if ((fp = hopen(infile, "r")) == NULL) {
print_error_errno("amplicon", "unable to open file %s.", infile);
return 1;
}
- char ref[1024];
while (line.l = 0, kgetline(&line, (kgets_func *)hgets, fp) >= 0) {
line_count++;
int hret;
- char strand;
if (line.l == 0 || *line.s == '#') continue;
if (strncmp(line.s, "track ", 6) == 0) continue;
if (strncmp(line.s, "browser ", 8) == 0) continue;
- if (get_strand) {
- if (sscanf(line.s, "%1023s %"SCNd64" %"SCNd64" %*s %*s %c",
- ref, &left, &right, &strand) != 4) {
- fprintf(samtools_stderr, "[amplicon] error: bad bed file format in line %d of %s.\n"
- "(N.B. ref/chrom name limited to 1023 characters.)\n",
- line_count, infile);
- ret = 1;
- goto error;
- }
- } else {
- if (sscanf(line.s, "%1023s %"SCNd64" %"SCNd64,
- ref, &left, &right) != 3) {
- fprintf(samtools_stderr, "[amplicon] error: bad bed file format in line %d of %s\n"
- "(N.B. ref/chrom name limited to 1023 characters.)\n",
- line_count, infile);
- ret = 1;
- goto error;
- }
+ // A list of the maximal number of columns we may want to parse.
+ // There may be more, but we don't use the ones beyond these.
+ const char *const scanf_str =
+ "%1023s %"SCNd64" %"SCNd64" %1023s %1023s %c";
+
+ // Extract the data from the line into the variables.
+ // Variables corresponding to any missing columns will remain
+ // uninitialised. We asked for all columns, but cols_parsed will
+ // return how many we found which can be validated against num_columns.
+ int cols_parsed = sscanf(line.s, scanf_str, ref,
+ &left, &right, name, score, &strand);
+ if (cols_parsed < (get_strand ? 6 : 3)) {
+ fprintf(samtools_stderr, "[amplicon] error: invalid bed file format in line %d of %s.\n"
+ "Parsed %d columns, but need at least %d\n"
+ "(N.B. ref/chrom name limited to 1023 characters.)\n",
+ line_count, infile, cols_parsed, get_strand ? 6 : 3);
+ ret = 1;
+ goto error;
}
bed_itr = kh_get(bed_list_hash, bed_lists, ref);
goto error;
}
+ if (ref_list) {
+ if (hts_resize(char **, num_refs + 1, &ref_list_sz, ref_list, 0) < 0) {
+ fprintf(samtools_stderr, "[amplicon] error: unable to allocate memory for ref name list.\n");
+ ret = 1;
+ goto error;
+ }
+ (*ref_list)[num_refs++] = ref_name;
+ }
bed_itr = kh_put(bed_list_hash, bed_lists, ref_name, &hret);
if (hret > 0) {
list = &kh_val(bed_lists, bed_itr);
}
+ // add the bed entry to the list, growing the list if necessary
if (list->length == list->size) {
bed_entry_t *tmp;
list->bp[list->length].left = left;
list->bp[list->length].right = right;
+ list->bp[list->length].name = NULL;
+ list->bp[list->length].score = NULL;
+ if (cols_parsed >= 4) {
+ list->bp[list->length].name = strdup(name);
+ if (list->bp[list->length].name == NULL) {
+ fprintf(samtools_stderr, "[amplicon] error: unable to allocate memory for name in line %d of %s: %s.\n", line_count, infile, line.s);
+ ret = 1;
+ goto error;
+ }
+ }
+ if (cols_parsed >= 5) {
+ list->bp[list->length].score = strdup(score);
+ if (list->bp[list->length].score == NULL) {
+ fprintf(samtools_stderr, "[amplicon] error: unable to allocate memory for score in line %d of %s: %s\n", line_count, infile, line.s);
+ ret = 1;
+ goto error;
+ }
+ }
+
+ list->bp[list->length].num_reads = 0;
if (get_strand) {
if (strand == '+') {
ret = 1;
}
+ if (num_refs_out)
+ *num_refs_out = num_refs;
+
error:
ks_free(&line);
for (itr = kh_begin(hash); itr != kh_end(hash); ++itr) {
if (kh_exist(hash, itr)) {
- free(kh_val(hash, itr).bp);
+ bed_entry_list_t list = kh_val(hash, itr);
+ for (int i = 0; i < list.length; i++) {
+ free(list.bp[i].name);
+ free(list.bp[i].score);
+ }
+ free(list.bp);
free((char *)kh_key(hash, itr));
kh_key(hash, itr) = NULL;
}
static int matching_clip_site(bed_entry_list_t *sites, hts_pos_t pos,
int is_rev, int use_strand, int64_t longest,
cl_param_t *param) {
- int i, size; // may need this to be variable
+ int i, size, used_i;
int tol = param->tol;
int l = 0, mid = sites->length / 2, r = sites->length;
int pos_tol = is_rev ? (pos > tol ? pos - tol : 0) : pos;
}
size = 0;
+ used_i = -1;
for (i = l; i < sites->length; i++) {
hts_pos_t mod_left, mod_right;
if (is_rev) {
if (size < pos - sites->bp[i].left) {
size = pos - sites->bp[i].left;
+ used_i = i;
}
} else {
if (size < sites->bp[i].right - pos) {
size = sites->bp[i].right - pos;
+ used_i = i;
}
}
}
}
-
+ if (used_i >= 0) {
+ sites->bp[used_i].num_reads++;
+ }
return size;
}
}
// Copy remaining QUAL
- memmove(new_qual, orig_qual, rec->core.l_qseq - qry_removed);
+ memmove(new_qual, orig_qual + qry_removed, rec->core.l_qseq - qry_removed);
// Set new l_qseq
rec_out->core.l_qseq -= qry_removed;
kstring_t seq = KS_INITIALIZE;
bed_entry_list_t *sites;
FILE *stats_fp = samtools_stderr;
+ FILE *bed_count_summary_fp = samtools_stderr;
khash_t(bed_list_hash) *bed_hash = kh_init(bed_list_hash);
+ char **bed_ref_list = NULL;
+ size_t num_bed_refs = 0;
- if (load_bed_file_multi_ref(bedfile, param->use_strand, 1, bed_hash)) {
+ if (load_bed_file_multi_ref(bedfile, param->use_strand, 1, bed_hash,
+ &bed_ref_list, &num_bed_refs)) {
fprintf(samtools_stderr, "[ampliconclip] error: unable to load bed file.\n");
goto fail;
}
if (file_open) {
fclose(stats_fp);
+ file_open = 0;
+ }
+
+ if (param->primer_counts_file) {
+ if ((bed_count_summary_fp = fopen(param->primer_counts_file, "w")) == NULL) {
+ fprintf(samtools_stderr, "[ampliconclip] warning: cannot write count summary to %s.\n", param->primer_counts_file);
+ } else {
+ file_open = 1;
+ }
+
+ //print out the number of reads for each bed entry, bedgraph format
+ fprintf(bed_count_summary_fp, "#CHR\tLEFT\tRIGHT\tNAME\tSCORE\tSTRAND\tNUM_CLIPPED\n");
+ size_t refidx;
+ for (refidx = 0; refidx < num_bed_refs; refidx++) {
+ khiter_t itr = kh_get(bed_list_hash, bed_hash, bed_ref_list[refidx]);
+ if (itr >= kh_end(bed_hash)) {
+ fprintf(samtools_stderr, "[ampliconclip] error: %s has gone missing from the hash table\n", bed_ref_list[refidx]);
+ goto fail;
+ }
+ sites = &kh_val(bed_hash, itr);
+ int i;
+ for (i = 0; i < sites->length; i++) {
+ char* strand_out = param->use_strand ? (sites->bp[i].rev ? "-" : "+") : ".";
+ fprintf(bed_count_summary_fp, "%s\t%"PRId64"\t%"PRId64"\t%s\t%s\t%s\t%"PRId64"\n",
+ kh_key(bed_hash, itr), sites->bp[i].left, sites->bp[i].right, sites->bp[i].name,
+ sites->bp[i].score, strand_out, sites->bp[i].num_reads);
+ }
+ }
+ if (file_open) {
+ fclose(bed_count_summary_fp);
+ }
}
ret = 0;
fail:
+ free(bed_ref_list);
destroy_bed_hash(bed_hash);
ks_free(&oat);
ks_free(&seq);
static void usage(void) {
fprintf(samtools_stderr, "Usage: samtools ampliconclip -b BED file <input.bam> -o <output.bam>\n\n");
fprintf(samtools_stderr, "Option: \n");
- fprintf(samtools_stderr, " -b FILE BED file of regions (eg amplicon primers) to be removed.\n");
- fprintf(samtools_stderr, " -o FILE output file name (default samtools_stdout).\n");
- fprintf(samtools_stderr, " -f FILE write stats to file name (default samtools_stderr)\n");
- fprintf(samtools_stderr, " -u Output uncompressed data\n");
- fprintf(samtools_stderr, " --soft-clip soft clip amplicon primers from reads (default)\n");
- fprintf(samtools_stderr, " --hard-clip hard clip amplicon primers from reads.\n");
- fprintf(samtools_stderr, " --both-ends clip on both 5' and 3' ends.\n");
- fprintf(samtools_stderr, " --strand use strand data from BED file to match read direction.\n");
- fprintf(samtools_stderr, " --clipped only output clipped reads.\n");
- fprintf(samtools_stderr, " --fail mark unclipped, mapped reads as QCFAIL.\n");
- fprintf(samtools_stderr, " --filter-len INT do not output reads INT size or shorter.\n");
- fprintf(samtools_stderr, " --fail-len INT mark as QCFAIL reads INT size or shorter.\n");
- fprintf(samtools_stderr, " --unmap-len INT unmap reads INT size or shorter, default 0.\n");
- fprintf(samtools_stderr, " --no-excluded do not write excluded reads (unmapped or QCFAIL).\n");
- fprintf(samtools_stderr, " --rejects-file FILE file to write filtered reads.\n");
- fprintf(samtools_stderr, " --original for clipped entries add an OA tag with original data.\n");
- fprintf(samtools_stderr, " --keep-tag for clipped entries keep the old NM and MD tags.\n");
- fprintf(samtools_stderr, " --tolerance match region within this number of bases, default 5.\n");
- fprintf(samtools_stderr, " --no-PG do not add an @PG line.\n");
+ fprintf(samtools_stderr, " -b FILE BED file of regions (eg amplicon primers) to be removed.\n");
+ fprintf(samtools_stderr, " -o FILE output file name (default: samtools_stdout).\n");
+ fprintf(samtools_stderr, " -f FILE write stats to file name (default: samtools_stderr)\n");
+ fprintf(samtools_stderr, " -u Output uncompressed data\n");
+ fprintf(samtools_stderr, " --soft-clip soft clip amplicon primers from reads (default)\n");
+ fprintf(samtools_stderr, " --hard-clip hard clip amplicon primers from reads.\n");
+ fprintf(samtools_stderr, " --both-ends clip on both 5' and 3' ends.\n");
+ fprintf(samtools_stderr, " --strand use strand data from BED file to match read direction.\n");
+ fprintf(samtools_stderr, " --clipped only output clipped reads.\n");
+ fprintf(samtools_stderr, " --fail mark unclipped, mapped reads as QCFAIL.\n");
+ fprintf(samtools_stderr, " --filter-len INT do not output reads INT size or shorter.\n");
+ fprintf(samtools_stderr, " --fail-len INT mark as QCFAIL reads INT size or shorter.\n");
+ fprintf(samtools_stderr, " --unmap-len INT unmap reads INT size or shorter, default 0.\n");
+ fprintf(samtools_stderr, " --no-excluded do not write excluded reads (unmapped or QCFAIL).\n");
+ fprintf(samtools_stderr, " --rejects-file FILE file to write filtered reads.\n");
+ fprintf(samtools_stderr, " --primer-counts FILE file to write read counts per bed entry (bedgraph format).\n");
+ fprintf(samtools_stderr, " --original for clipped entries add an OA tag with original data.\n");
+ fprintf(samtools_stderr, " --keep-tag for clipped entries keep the old NM and MD tags.\n");
+ fprintf(samtools_stderr, " --tolerance match region within this number of bases, default 5.\n");
+ fprintf(samtools_stderr, " --no-PG do not add an @PG line.\n");
sam_global_opt_help(samtools_stderr, "-.O..@-.");
fprintf(samtools_stderr, "\nAbout: Soft clips read alignments where they match BED file defined regions.\n"
"Default clipping is only on the 5' end.\n\n");
{"fail-len", required_argument, NULL, 1010},
{"no-excluded", no_argument, NULL, 1011},
{"rejects-file", required_argument, NULL, 1012},
- {"original", no_argument, NULL, 1013},
- {"keep-tag", no_argument, NULL, 1014},
- {"tolerance", required_argument, NULL, 1015},
- {"unmap-len", required_argument, NULL, 1016},
+ {"primer-counts", required_argument, NULL, 1013},
+ {"original", no_argument, NULL, 1014},
+ {"keep-tag", no_argument, NULL, 1015},
+ {"tolerance", required_argument, NULL, 1016},
+ {"unmap-len", required_argument, NULL, 1017},
{NULL, 0, NULL, 0}
};
case 1010: param.fail_len = atoi(optarg); break;
case 1011: param.unmapped = 1; break;
case 1012: param.rejects_file = optarg; break;
- case 1013: param.oa_tag = 1; break;
- case 1014: param.del_tag = 0; break;
- case 1015: param.tol = atoi(optarg); break;
- case 1016: param.unmap_len = atoi(optarg); break;
+ case 1013: param.primer_counts_file = optarg; break;
+ case 1014: param.oa_tag = 1; break;
+ case 1015: param.del_tag = 0; break;
+ case 1016: param.tol = atoi(optarg); break;
+ case 1017: param.unmap_len = atoi(optarg); break;
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
/* else fall-through */
case '?': usage(); samtools_exit(1);
typedef struct {
int64_t left;
int64_t right;
+ char *name;
+ char *score;
int rev;
+ int64_t num_reads;
} bed_entry_t;
typedef struct {
int size;
} bed_entry_list_t;
-KHASH_MAP_INIT_STR(bed_list_hash, bed_entry_list_t);
+KHASH_MAP_INIT_STR(bed_list_hash, bed_entry_list_t)
#define BED_LIST_INIT {NULL, 0, 0, 0, {0}}
int load_bed_file_multi_ref(char *infile, int get_strand,
- int sort_by_pos, khash_t(bed_list_hash) *bed_lists);
+ int sort_by_pos, khash_t(bed_list_hash) *bed_lists,
+ char ***ref_list, size_t *num_refs);
void destroy_bed_hash(khash_t(bed_list_hash) *hash);
/* bam_cat.c -- efficiently concatenates bam files.
- Copyright (C) 2008-2009, 2011-2013, 2015-2017, 2019, 2021 Genome Research Ltd.
+ Copyright (C) 2008-2009, 2011-2013, 2015-2017, 2019, 2021,
+ 2023-2024 Genome Research Ltd.
Modified SAMtools work copyright (C) 2010 Illumina, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
#include "htslib/sam.h"
#include "htslib/cram.h"
#include "htslib/kstring.h"
+#include "htslib/hfile.h"
#include "samtools.h"
#include "sam_opts.h"
-/*
- * Check the files are consistent and capable of being concatenated.
- * Also fills out the version numbers and produces a new sam_hdr_t
- * structure with merged RG lines.
- * Note it is only a simple merge.
- *
- * Returns updated header on success;
- * NULL on failure.
- */
-static sam_hdr_t *cram_cat_check_hdr(int nfn, char * const *fn, const sam_hdr_t *h,
- int *vers_maj_p, int *vers_min_p) {
+/// cat_check_merge_hdr - check compatibility and merge RG hearders merges RGon both CRAM and BAM.
+/** @param firstfile - pointer to the 1sr file opened in caller
+ * @param nfn - number of files to be processed, including the firstfile
+ * @param fn - array of file paths to be processed
+ * @param h - sam header pointer which contains explicitly given header
+ * @param vers_maj_p - cram major version set and send out for output creation
+ * @param vers_min_p - cram min version set and send out for output creation
+ * @param out_h - pointer to sam header pointer, outputs the merged header
+ * returns array of opened samFile pointers on success and NULL on failure
+ * This method has the merged header processing for cram and bam.
+ * RG lines are merged for both cram and bam. For cram, version match for each
+ * file and order match of RG lines are compared as well.
+ * Note: it is a simple merge of RG lines alone.
+*/
+static samFile** cat_check_merge_hdr(samFile * const firstfile, int nfn, char * const *fn, const sam_hdr_t *h,
+ int *vers_maj_p, int *vers_min_p, sam_hdr_t **out_h) {
int i, vers_maj = -1, vers_min = -1;
sam_hdr_t *new_h = NULL, *old_h = NULL;
samFile *in = NULL;
kstring_t ks = KS_INITIALIZE;
-
- if (h) {
- new_h = sam_hdr_dup(h);
- if (!new_h) {
- fprintf(stderr, "[%s] ERROR: header duplication failed.\n",
- __func__);
- goto fail;
+ samFile **files = calloc(nfn, sizeof(samFile *));
+ if(!files) {
+ fprintf(stderr, "[%s] ERROR: failed to allocate space for file handles.\n", __func__);
+ return NULL;
+ }
+ if (!out_h || !firstfile) {
+ fprintf(stderr, "[%s] ERROR: header check failed.\n", __func__);
+ goto fail;
+ }
+ if (*out_h) { //use header if one is already present
+ new_h = *out_h;
+ }
+ else {
+ if (h) { //use the explicit header given
+ new_h = sam_hdr_dup(h);
+ if (!new_h) {
+ fprintf(stderr, "[%s] ERROR: header duplication failed.\n",
+ __func__);
+ goto fail;
+ }
}
}
for (i = 0; i < nfn; ++i) {
- cram_fd *in_c;
int ki;
-
- in = sam_open(fn[i], "rc");
+ //1st file is already open and passed, rest open locally
+ files[i] = in = i ? sam_open(fn[i], "r") : firstfile;
if (in == 0) {
print_error_errno("cat", "fail to open file '%s'", fn[i]);
goto fail;
}
- in_c = in->fp.cram;
-
- int vmaj = cram_major_vers(in_c);
- int vmin = cram_minor_vers(in_c);
- if ((vers_maj != -1 && vers_maj != vmaj) ||
- (vers_min != -1 && vers_min != vmin)) {
- fprintf(stderr, "[%s] ERROR: input files have differing version numbers.\n",
- __func__);
+ if (firstfile->format.format != in->format.format) {
+ print_error("cat", "File %s is of different format!", fn[i]);
goto fail;
}
- vers_maj = vmaj;
- vers_min = vmin;
+ if (firstfile->format.format == cram) { //version check for cram
+ cram_fd *in_c;
+ in_c = in->fp.cram;
+
+ int vmaj = cram_major_vers(in_c);
+ int vmin = cram_minor_vers(in_c);
+ if ((vers_maj != -1 && vers_maj != vmaj) ||
+ (vers_min != -1 && vers_min != vmin)) {
+ fprintf(stderr, "[%s] ERROR: input files have differing version numbers.\n",
+ __func__);
+ goto fail;
+ }
+ vers_maj = vmaj;
+ vers_min = vmin;
+ }
old_h = sam_hdr_read(in);
if (!old_h) {
goto fail;
}
sam_hdr_destroy(old_h);
- sam_close(in);
+ old_h = NULL;
continue;
}
-
+ //merge RG lines
int old_count = sam_hdr_count_lines(old_h, "RG");
for (ki = 0; ki < old_count; ki++) {
const char *old_name = sam_hdr_line_name(old_h, "RG", ki);
}
}
- if (old_count > 1 && sam_hdr_count_lines(new_h, "RG") == old_count) {
+ if (firstfile->format.format == cram && old_count > 1 && sam_hdr_count_lines(new_h, "RG") == old_count) {
+ //RG order check for cram
for (ki = 0; ki < old_count; ki++) {
const char *old_name = sam_hdr_line_name(old_h, "RG", ki);
const char *new_name = sam_hdr_line_name(new_h, "RG", ki);
}
}
- sam_hdr_destroy(old_h);
- sam_close(in);
+ sam_hdr_destroy(old_h); old_h = NULL;
}
ks_free(&ks);
- *vers_maj_p = vers_maj;
- *vers_min_p = vers_min;
-
- return new_h;
+ if (vers_maj_p) {
+ *vers_maj_p = vers_maj;
+ }
+ if (vers_min_p) {
+ *vers_min_p = vers_min;
+ }
+ *out_h = new_h;
+ return files;
fail:
ks_free(&ks);
if (old_h) sam_hdr_destroy(old_h);
if (new_h) sam_hdr_destroy(new_h);
- if (in) sam_close(in);
+ *out_h = NULL;
+ for (i = 1; i < nfn; ++i) { //close files other than the firstfile
+ if (files[i]) {
+ sam_close(files[i]);
+ }
+ }
+ free(files);
return NULL;
}
-/*
- * CRAM files don't store the RG:Z:ID per read in the aux field.
- * Instead they have a numerical data series (RG) to point each read
- * back to the Nth @RG line in the file. This means that we may need
- * to edit the RG data series (if the files were produced from
- * "samtools split" for example).
- *
- * The encoding method is stored in the compression header. Typical
- * examples:
- *
- * RG => EXTERNAL {18} # Block content-id 18 holds RG values
- * # as a series of ITF8 encoded values
- *
- * RG => HUFFMAN {1, 255, 255, 255, 255, 255, 1, 0}
- * # One RG value #-1. (No RG)
- *
- * RG => HUFFMAN {1, 0, 1, 0} # One RG value #0 (always first RG)
- *
- * RG => HUFFMAN {2, 0, 1, 2, 1, 1}
- * # Two RG values, #0 and #1, written
- * # to the CORE block and possibly
- * # mixed with other data series.
- *
- * A single value can (but may not be) implemented as a zero bit
- * huffman code. In this situation we can change the meta-data in the
- * compression header to renumber an RG value..
+/* ----------------------------------------------------------------------
+ * CRAM cat
*/
-int cram_cat(int nfn, char * const *fn, const sam_hdr_t *h, const char* outcram, sam_global_args *ga, char *arg_list, int no_pg)
+
+// Reports the number of CRAM containers spanning a specified region if
+// specified, or the entire file if not.
+// This is the implements the "samtools cat -q [-r reg]" functionality.
+//
+// Returns 0 on success, <0 on error.
+static int cram_query_ncont(int nfn, char * const *fn, char *reg) {
+ int i;
+ hts_idx_t *idx = NULL;
+ sam_hdr_t *hdr = NULL;
+ hts_itr_t *iter = NULL;
+ samFile *in = NULL;
+
+ for (i = 0; i < nfn; i++) {
+ in = sam_open(fn[i], "r");
+ if (!in) {
+ print_error_errno("cat", "Couldn't open file %s", fn[i]);
+ return -1;
+ }
+ idx = sam_index_load(in, fn[i]);
+ if (!idx) {
+ print_error("cat", "No index found");
+ goto err;
+ }
+
+ off_t cstart = 0, cend = 0;
+ if (reg) {
+ sam_hdr_t *hdr = sam_hdr_read(in);
+ hts_itr_t *iter = sam_itr_querys(idx, hdr, reg);
+ if (!hdr) {
+ print_error("cat", "Unable to read header");
+ goto err;
+ }
+ if (!iter) {
+ print_error("cat", "Unable to parse region");
+ goto err;
+ }
+
+ if (cram_index_extents(in->fp.cram, iter->tid, iter->beg,
+ iter->end, &cstart, &cend) < 0) {
+ print_error("cat", "Failed to query index");
+ goto err;
+ }
+
+ hts_itr_destroy(iter);
+ sam_hdr_destroy(hdr);
+ }
+
+ int64_t first, last;
+ int64_t nc = cram_num_containers_between(in->fp.cram, cstart, cend,
+ &first, &last);
+ printf("%s\t%"PRId64"\t%"PRId64"\t%"PRId64"\n",
+ fn[i], nc, first, last);
+ sam_close(in);
+ hts_idx_destroy(idx);
+ }
+
+ return 0;
+
+ err:
+ if (idx)
+ hts_idx_destroy(idx);
+ if (hdr)
+ sam_hdr_destroy(hdr);
+ if (iter)
+ hts_itr_destroy(iter);
+
+ sam_close(in);
+
+ return -1;
+}
+
+// Container range #:A-B or #:A.
+// Returns 0 on success, -1 on failure.
+static int cram_handle_cnum_region(cram_fd *fd, hts_idx_t *idx,
+ char *reg, off_t *cstart, off_t *cend) {
+ int cnum_start, cnum_end; // container versions
+ int n = sscanf(reg, "#:%d-%d", &cnum_start, &cnum_end);
+
+ if (n == 0) {
+ print_error("cat", "ERROR: Malformed region: %s", reg);
+ return -1;
+ } else if (n == 1) {
+ cnum_end = cnum_start;
+ }
+
+ int64_t nc = cram_num_containers(fd);
+ if (cnum_end >= nc) {
+ print_error("cat", "Too many containers. "
+ "The end range should be < %"PRId64, nc);
+ return -1;
+ }
+
+ // Container number to offset
+ *cstart = cram_container_num2offset(fd, cnum_start);
+ *cend = cram_container_num2offset(fd, cnum_end);
+ if (*cstart < 0 || *cend < 0)
+ return -1;
+
+ // Seek manually
+ return cram_seek(fd, *cstart, SEEK_SET);
+}
+
+// Normal range chr:start-end.
+// Returns an hts iterator on success, NULL on failure.
+static hts_itr_t *cram_handle_region(cram_fd *fd, hts_idx_t *idx, sam_hdr_t *h,
+ char *reg, off_t *cstart, off_t *cend) {
+ hts_itr_t *iter;
+
+ if (!idx) {
+ fprintf(stderr, "[%s] ERROR: No index found.\n", __func__);
+ return NULL;
+ }
+
+ // This does an implicit seek and modifies the cram_fd.
+ if (!(iter = sam_itr_querys(idx, h, reg))) {
+ print_error("cat", "Unable to parse region %s", reg);
+ return NULL;
+ }
+ if (cram_index_extents(fd, iter->tid, iter->beg,
+ iter->end, cstart, cend) < 0) {
+ print_error("cat", "Failed to query index");
+ return NULL;
+ }
+
+ return iter;
+}
+
+// Handle the -p A/B option to subdivide our region or file into portions.
+// Updates cstart/cend.
+// returns 0 on success (do something),
+// 1 on success (but nothing to do),
+// -1 on failure.
+static int cram_subdivide_part(cram_fd *fd, hts_idx_t *idx, char *part,
+ off_t *cstart, off_t *cend) {
+ int a, b;
+ // Part N of M
+ if (sscanf(part, "%d/%d", &a, &b) != 2) {
+ print_error("cat", "malformed region %s. Should be e.g. '1/10'", part);
+ return -1;
+ }
+
+ // Inclusive container numbers for range, 0 to NC-1
+ // Our part N/M is in container percentages as we can't have
+ // partial containers, so convert to that first.
+ int64_t cnum1, cnum2;
+ if (cstart) {
+ cnum1 = cram_container_offset2num(fd, *cstart);
+ cnum2 = cram_container_offset2num(fd, *cend);
+ } else {
+ cnum1 = 0;
+ cnum2 = cram_num_containers(fd)-1;
+ }
+
+ // Subdivide cnum1/cnum2 container numbers to new range cnum_start/end
+ int64_t nc = cnum2 - cnum1 + 1;
+ if (b > nc)
+ b = nc;
+
+ int cnum_start = (a-1)*(double)nc/b;
+ int cnum_end = a*(double)nc/b - 1;
+ if (cnum_start < 0 || cnum_end >= nc)
+ return 1;
+
+ // Then convert back to file offsets so we can seek and do htell
+ // to detect EOR/EOF.
+ *cstart = cram_container_num2offset(fd, cnum_start + cnum1);
+ *cend = cram_container_num2offset(fd, cnum_end + cnum1);
+
+ return 0;
+}
+
+// The main cram_cat interface.
+// Returns 0 on success, < 0 on error.
+int cram_cat(samFile * const firstfile, int nfn, char * const *fn,
+ const sam_hdr_t *h, const char* outcram, sam_global_args *ga,
+ char *arg_list, int no_pg, char *reg, char *part, int fast_reg)
{
- samFile *out;
+ samFile *out = NULL;
cram_fd *out_c;
- int i, vers_maj, vers_min;
+ int i, vers_maj, vers_min, ret = -1;
sam_hdr_t *new_h = NULL;
-
- /* Check consistent versioning and compatible headers */
- if (!(new_h = cram_cat_check_hdr(nfn, fn, h, &vers_maj, &vers_min)))
+ samFile **files = NULL;
+ hts_idx_t *idx = NULL;
+ hts_itr_t *iter = NULL;
+ sam_hdr_t *old_h = NULL;
+
+ // Check consistent versioning and compatible headers;
+ // merges RG lines, opens all files and returns them that multiple
+ // non-seekable stream inputs can be handled
+ if (!(files = cat_check_merge_hdr(firstfile, nfn, fn, h, &vers_maj,
+ &vers_min, &new_h)))
return -1;
+ if (!new_h) {
+ print_error_errno("cat", "failed to make output header");
+ goto closefiles;
+ }
+
/* Open the file with cram_vers */
char vers[100];
- sprintf(vers, "%d.%d", vers_maj, vers_min);
- out = sam_open_format(outcram, "wc", &ga->out);
+ snprintf(vers, sizeof(vers), "%d.%d", vers_maj, vers_min);
+
+ // Can override level=1 with e.g. "--output-fmt-option level=9"
+ out = sam_open_format(outcram, "wc1", &ga->out);
if (out == 0) {
print_error_errno("cat", "fail to open output file '%s'", outcram);
- return -1;
+ goto closefiles;
}
out_c = out->fp.cram;
cram_set_option(out_c, CRAM_OPT_VERSION, vers);
- //fprintf(stderr, "Creating cram vers %s\n", vers);
if (!no_pg && sam_hdr_add_pg(new_h, "samtools",
"VN", samtools_version(),
arg_list ? "CL": NULL,
arg_list ? arg_list : NULL,
NULL))
- return -1;
+ goto closefiles;
if (sam_hdr_write(out, new_h) < 0) {
print_error_errno("cat", "Couldn't write header");
- return -1;
+ goto closefiles;
}
+ out_c = out->fp.cram;
for (i = 0; i < nfn; ++i) {
samFile *in;
cram_fd *in_c;
cram_container *c;
- sam_hdr_t *old_h;
int new_rg = -1;
- in = sam_open(fn[i], "rc");
+ in = files[i];
if (in == 0) {
print_error_errno("cat", "fail to open file '%s'", fn[i]);
- return -1;
+ goto closefiles;
}
in_c = in->fp.cram;
old_h = sam_hdr_read(in);
if (!old_h) {
print_error("cat", "fail to read the header of file '%s'", fn[i]);
- return -1;
+ goto closefiles;
}
// Compute RG mapping if suitable for changing.
new_rg = sam_hdr_line_index(new_h, "RG", old_name);
if (new_rg < 0) {
print_error("cat", "fail to find @RG line '%s' in the new header", old_name);
- return -1;
+ goto closefiles;
}
} else {
print_error("cat", "fail to find @RG line in file '%s'", fn[i]);
- return -1;
+ goto closefiles;
}
} else {
new_rg = 0;
}
+ // We have multiple region syntax. Either the standard chr:start-end
+ // or a cat-specific #:num-num for explicit container numbers.
+ // Both of these seek to a specific file offset and also have the
+ // end offset known so we can use htell to detect when we're done.
+ // Those offsets are in cstart and cend.
+ //
+ // However for the e.g. -p 1/10 syntax we need to know the container
+ // numbers corresponding to the cstart/cend offsets as we can't
+ // start half way through a container when doing fractions.
+ // These are cnum1 and cnum2. (We could short cut this for the
+ // #:num-num, but it's simpler to just treat all regions identically.)
+ off_t cstart = 0, cend = 0;
+ int filter_by_cnum = 0;
+
+ if (reg || part) {
+ idx = sam_index_load(in, fn[i]);
+ if (!idx) {
+ print_error("cat", "failed to load index");
+ goto closefiles;
+ }
+ }
+
+ if (reg) {
+ if (strncmp(reg, "#:", 2) == 0) {
+ // Region as container numbers
+ if (cram_handle_cnum_region(in_c, idx, reg, &cstart,&cend) < 0)
+ goto closefiles;
+
+ filter_by_cnum = 1;
+ } else {
+ // Normal range chr:start-end
+ if (!(iter = cram_handle_region(in_c, idx, old_h, reg,
+ &cstart, &cend)))
+ goto closefiles;
+ }
+ }
+
+ // We can also take a range above and subdivide it into parts.
+ // Eg -r chr1 -p 1/10 (... to -p 10/10). Part only implies
+ // portions of the entire file.
+ if (part) {
+ int r = cram_subdivide_part(in_c, idx, part, &cstart, &cend);
+ if (r != 0) {
+ if (r > 0) // Not an error, just nothing to do
+ ret = 0;
+ goto closefiles;
+ }
+ }
+
+ if (cstart) // reg or part
+ if (0 != cram_seek(in_c, cstart, SEEK_SET))
+ goto closefiles;
+
+
+ // Make refid -2 ("*") come after other chromosomes, for easy sort
+ int itid = iter
+ ? (iter->tid == HTS_IDX_NOCOOR ? INT_MAX : iter->tid)
+ : 0;
+ int last_ref_id = -99;
+
+ off_t before_hdr = htell(cram_fd_get_fp(in_c));
+
// Copy contains and blocks within them
while ((c = cram_read_container(in_c))) {
if (cram_container_is_empty(in_c)) {
cram_block *blk;
// Container compression header
if (!(blk = cram_read_block(in_c)))
- return -1;
+ goto closefiles;
cram_free_block(blk);
cram_free_container(c);
continue;
}
+ int filter = 0;
+
// If we have just one RG key and new_rg != 0 then
// we need to edit the compression header. IF WE CAN.
if (new_rg) {
+ if (reg) {
+ print_error("cat", "Cannot specify a region while "
+ "transcoding RG lines");
+ goto closefiles;
+ }
int zero = 0;
//fprintf(stderr, "Transcode RG %d to %d\n", 0, new_rg);
cram_transcode_rg(in_c, out_c, c, 1, &zero, &new_rg);
} else {
int32_t num_slices;
- cram_block *blk;
- // Not switching rg so do the usual read/write loop
- if (cram_write_container(out_c, c) != 0)
- return -1;
+ if (reg) {
+ if (before_hdr > cend) {
+ cram_free_container(c);
+ break;
+ }
+ }
- // Container compression header
- if (!(blk = cram_read_block(in_c)))
- return -1;
- if (cram_write_block(out_c, blk) != 0) {
- cram_free_block(blk);
- return -1;
+ // For chr:start-end regions, do we need to filter or skip?
+ if (iter && reg) {
+ int refid;
+ hts_pos_t start, span, end;
+
+ cram_container_get_coords(c, &refid, &start, &span);
+ end = start+span;
+
+ // Make refid -1 ("*") come after other chromosomes
+ if (refid == -1)
+ refid = INT_MAX;
+
+ if (refid > itid || start > iter->end) {
+ // Beyond the requested range
+ break;
+ } else if (refid == -2) {
+ // Multi-ref containers. We only support this if
+ // the RI data series is in a container by itself.
+ filter = 3;
+ } else if (refid < itid || end < iter->beg) {
+ // Skip, in case of mixed size containers.
+ // Eg: Use "=", skip "-".
+ // ==========|======|=
+ // ---- ==|== ==|== ----
+ // ----- | ==== ----
+ filter = 2;
+ } else if (start < iter->beg || end > iter->end) {
+ // Container overlaps region.
+ // Fast mode just copies overlapping containers.
+ // Slow mode does a precise filtering by reading and
+ // filtering each record in turn so it's the same as
+ // a samtools view command.
+ filter = fast_reg ? 0 : 1;
+ }
+ // else we're in an "internal" container, so just copy
}
- cram_free_block(blk);
+ if (filter && last_ref_id == -1 && itid == INT_MAX)
+ // Multi-ref containers consisting solely of ref "*" are
+ // common, but if it's sorted then we know it's "*" from
+ // here on so we don't need to filter despite multi-ref.
+ filter = 0;
- // Container num_blocks can be invalid, due to a bug.
- // Instead we iterate in slice context instead.
- (void)cram_container_get_landmarks(c, &num_slices);
- cram_copy_slice(in_c, out_c, num_slices);
- }
+ if (filter) {
+ // Filter or skip
+ cram_filter_container(in_c, out_c, c, &last_ref_id);
+ } else {
+ // Copy. Consider adding a cram_copy_container API instead.
+ // Container compression header
+ cram_block *blk;
+ if (!(blk = cram_read_block(in_c)))
+ goto closefiles;
+
+ // Not switching rg so do the usual read/write loop
+ if (cram_write_container(out_c, c) != 0)
+ goto closefiles;
+
+ // Contatiner compression header
+ if (cram_write_block(out_c, blk) != 0) {
+ cram_free_block(blk);
+ goto closefiles;
+ }
+
+ // Container num_blocks can be invalid, due to a bug.
+ // Instead we iterate in slice context instead.
+ (void)cram_container_get_landmarks(c, &num_slices);
+ if (cram_copy_slice(in_c, out_c, num_slices) < 0) {
+ cram_free_block(blk);
+ goto closefiles;
+ }
+ cram_free_block(blk);
+ }
+ }
cram_free_container(c);
- }
+ // Location of next container start
+ before_hdr = htell(cram_fd_get_fp(in_c));
+ if (filter_by_cnum && before_hdr > cend)
+ break;
+ }
sam_hdr_destroy(old_h);
- sam_close(in);
+ old_h = NULL;
+
+ if (idx) {
+ hts_idx_destroy(idx);
+ idx = NULL;
+ }
+
+ if (iter) {
+ hts_itr_destroy(iter);
+ iter = NULL;
+ }
}
- sam_close(out);
- sam_hdr_destroy(new_h);
+ ret = 0;
- return 0;
-}
+closefiles:
+ if (old_h)
+ sam_hdr_destroy(old_h);
+
+ if (idx)
+ hts_idx_destroy(idx);
+
+ if (iter)
+ hts_itr_destroy(iter);
+ if (out)
+ sam_close(out);
+
+ if (new_h)
+ sam_hdr_destroy(new_h);
+
+ for (i = 1; i < nfn; ++i) { //skip firstfile and close rest
+ if (files[i]) {
+ sam_close(files[i]);
+ }
+ }
+ free(files);
+ return ret;
+}
+/* ----------------------------------------------------------------------
+ * BAM cat
+ */
#define BUF_SIZE 0x10000
#define GZIPID1 31
#define BGZF_EMPTY_BLOCK_SIZE 28
-int bam_cat(int nfn, char * const *fn, sam_hdr_t *h, const char* outbam, char *arg_list, int no_pg)
+int bam_cat(samFile * const firstfile, int nfn, char * const *fn, sam_hdr_t *h, const char* outbam, char *arg_list, int no_pg)
{
- BGZF *fp, *in = NULL;
+ BGZF *fp = NULL, *in = NULL;
uint8_t *buf = NULL;
uint8_t ebuf[BGZF_EMPTY_BLOCK_SIZE];
const int es=BGZF_EMPTY_BLOCK_SIZE;
int i;
+ samFile **files = NULL;
+ sam_hdr_t *new_h = NULL;
+ /* merges RG lines, opens all files and returns them that multiple non-seekable
+ stream inputs can be handled */
+ if (!(files = cat_check_merge_hdr(firstfile, nfn, fn, h, NULL, NULL, &new_h)))
+ return -1;
+ if (!new_h) {
+ print_error_errno("cat", "failed to make output header");
+ goto fail;
+ }
fp = strcmp(outbam, "-")? bgzf_open(outbam, "w") : bgzf_fdopen(fileno(stdout), "w");
if (fp == 0) {
print_error_errno("cat", "fail to open output file '%s'", outbam);
- return -1;
+ goto fail;
}
- if (h) {
- if (!no_pg && sam_hdr_add_pg(h, "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL))
- goto fail;
- if (bam_hdr_write(fp, h) < 0) {
- print_error_errno("cat", "Couldn't write header");
- goto fail;
- }
+ if (!no_pg && sam_hdr_add_pg(new_h, "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL))
+ goto fail;
+
+ if (bam_hdr_write(fp, new_h) < 0) {
+ print_error_errno("cat", "Couldn't write header");
+ goto fail;
}
buf = (uint8_t*) malloc(BUF_SIZE);
goto fail;
}
for(i = 0; i < nfn; ++i){
- sam_hdr_t *old;
int len,j;
-
- in = strcmp(fn[i], "-")? bgzf_open(fn[i], "r") : bgzf_fdopen(fileno(stdin), "r");
+ in = files[i]->fp.bgzf;
if (in == 0) {
print_error_errno("cat", "fail to open file '%s'", fn[i]);
goto fail;
}
- if (in->is_write) return -1;
-
- old = bam_hdr_read(in);
- if (old == NULL) {
- fprintf(stderr, "[%s] ERROR: couldn't read header for '%s'.\n",
- __func__, fn[i]);
- goto fail;
- }
- if (h == 0 && i == 0) {
- if (!no_pg && sam_hdr_add_pg(old, "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL))
- goto fail;
-
- if (bam_hdr_write(fp, old) < 0) {
- print_error_errno("cat", "Couldn't write header");
- goto fail;
- }
- }
+ if (in->is_write) goto fail;
if (in->block_offset < in->block_length) {
if (bgzf_write(fp, (char *)in->uncompressed_block + in->block_offset, in->block_length - in->block_offset) < 0) goto write_fail;
if (bgzf_raw_write(fp, ebuf, es) < 0) goto write_fail;
}
}
- sam_hdr_destroy(old);
- bgzf_close(in);
in = NULL;
}
free(buf);
if (bgzf_close(fp) < 0) {
fprintf(stderr, "[%s] Error on closing '%s'.\n", __func__, outbam);
- return -1;
+ goto fail;
+ }
+ for (i = 1; i < nfn; ++i) { //skip firstfile and close rest
+ if (files[i]) {
+ sam_close(files[i]);
+ }
}
+ free(files);
+ sam_hdr_destroy(new_h);
return 0;
write_fail:
fprintf(stderr, "[%s] Error writing to '%s'.\n", __func__, outbam);
fail:
- if (in) bgzf_close(in);
+ if (new_h) {
+ sam_hdr_destroy(new_h);
+ }
if (fp) bgzf_close(fp);
free(buf);
+
+ if (files) {
+ for(i = 1; i < nfn; ++i) { //except the firstfile
+ if(files[i]) {
+ sam_close(files[i]);
+ }
+ }
+ free(files);
+ }
return -1;
}
-
int main_cat(int argc, char *argv[])
{
sam_hdr_t *h = 0;
char *outfn = 0;
char **infns = NULL; // files to concatenate
int infns_size = 0;
- int c, ret = 0, no_pg = 0, usage = 0;
+ int c, ret = 0, no_pg = 0, usage = 0, query_ncont = 0, fast_mode = 0;
samFile *in;
sam_global_args ga;
+ char *reg = NULL, *part = NULL;
static const struct option lopts[] = {
- SAM_OPT_GLOBAL_OPTIONS('-', '-', '-', 0, '-', '@'),
+ SAM_OPT_GLOBAL_OPTIONS('-', '-', '-', 0, '-', '-'),
{"no-PG", no_argument, NULL, 1},
{ NULL, 0, NULL, 0 }
};
char *arg_list = NULL;
sam_global_args_init(&ga);
-
- while ((c = getopt_long(argc, argv, "h:o:b:@:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "h:o:b:r:p:qf", lopts, NULL)) >= 0) {
switch (c) {
case 'h': {
samFile *fph = sam_open(optarg, "r");
if (fph == 0) {
fprintf(stderr, "[%s] ERROR: fail to read the header from '%s'.\n", __func__, optarg);
- return 1;
+ ret = 1;
+ goto end;
}
h = sam_hdr_read(fph);
if (h == NULL) {
fprintf(stderr,
"[%s] ERROR: failed to read the header from '%s'.\n",
__func__, optarg);
- return 1;
+ ret = 1;
+ goto end;
}
sam_close(fph);
break;
case 1:
no_pg = 1;
break;
+ case 'r':
+ reg = optarg;
+ break;
+ case 'p':
+ part = optarg;
+ break;
+ case 'f':
+ fast_mode = 1;
+ break;
+ case 'q':
+ query_ncont=1;
+ break;
+
default:
if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
/* else fall-through */
if (!no_pg && !(arg_list = stringify_argv(argc+1, argv-1))) {
print_error("cat", "failed to create arg_list");
- return 1;
+ ret = 1;
+ goto end;
}
// Append files specified in argv to the list.
fprintf(stderr, " -h FILE copy the header from FILE [default is 1st input file]\n");
fprintf(stderr, " -o FILE output BAM/CRAM\n");
fprintf(stderr, " --no-PG do not add a PG line\n");
- sam_global_opt_help(stderr, "--..-@-.");
- return 1;
+ fprintf(stderr, "\nCRAM only options for filtering:\n");
+ fprintf(stderr, " -r REG filter to region REG.\n");
+ fprintf(stderr, " REG can also be #:cstart-cend for specific container numbers\n");
+ fprintf(stderr, " -p N/M Specify part N of M (where N is 1 to M inclusive)\n");
+ fprintf(stderr, " -f Fast mode: don't filter containers to exactly match region\n");
+ fprintf(stderr, " -q Query the total number of indexed containers\n");
+ fprintf(stderr, "\nStandard options:\n");
+ sam_global_opt_help(stderr, "---.---.");
+ ret = 1;
+ goto end;
}
in = sam_open(infns[0], "r");
if (!in) {
print_error_errno("cat", "failed to open file '%s'", infns[0]);
- return 1;
+ ret = 1;
+ goto end;
}
switch (hts_get_format(in)->format) {
case bam:
- sam_close(in);
- if (bam_cat(infns_size+nargv_fns, infns, h, outfn? outfn : "-", arg_list, no_pg) < 0)
+ if (bam_cat(in, infns_size+nargv_fns, infns, h, outfn? outfn : "-", arg_list, no_pg) < 0)
ret = 1;
break;
case cram:
- sam_close(in);
- if (cram_cat(infns_size+nargv_fns, infns, h, outfn? outfn : "-", &ga, arg_list, no_pg) < 0)
+ if (query_ncont) {
+ if (cram_query_ncont(infns_size+nargv_fns, infns, reg) < 0)
+ ret = 1;
+ } else {
+ if (cram_cat(in, infns_size+nargv_fns, infns, h,
+ outfn? outfn : "-", &ga, arg_list, no_pg, reg,
+ part, fast_mode) < 0)
ret = 1;
+ }
break;
default:
- sam_close(in);
fprintf(stderr, "[%s] ERROR: input is not BAM or CRAM\n", __func__);
- return 1;
+ ret = 1;
}
+ sam_close(in);
end:
if (infns_size > 0) {
free(arg_list);
if (h)
sam_hdr_destroy(h);
+ sam_global_args_free(&ga);
return ret;
}
/* bam_cat.c -- efficiently concatenates bam files.
- Copyright (C) 2008-2009, 2011-2013, 2015-2017, 2019, 2021 Genome Research Ltd.
+ Copyright (C) 2008-2009, 2011-2013, 2015-2017, 2019, 2021,
+ 2023-2024 Genome Research Ltd.
Modified SAMtools work copyright (C) 2010 Illumina, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
#include "htslib/sam.h"
#include "htslib/cram.h"
#include "htslib/kstring.h"
+#include "htslib/hfile.h"
#include "samtools.h"
#include "sam_opts.h"
-/*
- * Check the files are consistent and capable of being concatenated.
- * Also fills out the version numbers and produces a new sam_hdr_t
- * structure with merged RG lines.
- * Note it is only a simple merge.
- *
- * Returns updated header on success;
- * NULL on failure.
- */
-static sam_hdr_t *cram_cat_check_hdr(int nfn, char * const *fn, const sam_hdr_t *h,
- int *vers_maj_p, int *vers_min_p) {
+/// cat_check_merge_hdr - check compatibility and merge RG hearders merges RGon both CRAM and BAM.
+/** @param firstfile - pointer to the 1sr file opened in caller
+ * @param nfn - number of files to be processed, including the firstfile
+ * @param fn - array of file paths to be processed
+ * @param h - sam header pointer which contains explicitly given header
+ * @param vers_maj_p - cram major version set and send out for output creation
+ * @param vers_min_p - cram min version set and send out for output creation
+ * @param out_h - pointer to sam header pointer, outputs the merged header
+ * returns array of opened samFile pointers on success and NULL on failure
+ * This method has the merged header processing for cram and bam.
+ * RG lines are merged for both cram and bam. For cram, version match for each
+ * file and order match of RG lines are compared as well.
+ * Note: it is a simple merge of RG lines alone.
+*/
+static samFile** cat_check_merge_hdr(samFile * const firstfile, int nfn, char * const *fn, const sam_hdr_t *h,
+ int *vers_maj_p, int *vers_min_p, sam_hdr_t **out_h) {
int i, vers_maj = -1, vers_min = -1;
sam_hdr_t *new_h = NULL, *old_h = NULL;
samFile *in = NULL;
kstring_t ks = KS_INITIALIZE;
-
- if (h) {
- new_h = sam_hdr_dup(h);
- if (!new_h) {
- fprintf(samtools_stderr, "[%s] ERROR: header duplication failed.\n",
- __func__);
- goto fail;
+ samFile **files = calloc(nfn, sizeof(samFile *));
+ if(!files) {
+ fprintf(samtools_stderr, "[%s] ERROR: failed to allocate space for file handles.\n", __func__);
+ return NULL;
+ }
+ if (!out_h || !firstfile) {
+ fprintf(samtools_stderr, "[%s] ERROR: header check failed.\n", __func__);
+ goto fail;
+ }
+ if (*out_h) { //use header if one is already present
+ new_h = *out_h;
+ }
+ else {
+ if (h) { //use the explicit header given
+ new_h = sam_hdr_dup(h);
+ if (!new_h) {
+ fprintf(samtools_stderr, "[%s] ERROR: header duplication failed.\n",
+ __func__);
+ goto fail;
+ }
}
}
for (i = 0; i < nfn; ++i) {
- cram_fd *in_c;
int ki;
-
- in = sam_open(fn[i], "rc");
+ //1st file is already open and passed, rest open locally
+ files[i] = in = i ? sam_open(fn[i], "r") : firstfile;
if (in == 0) {
print_error_errno("cat", "fail to open file '%s'", fn[i]);
goto fail;
}
- in_c = in->fp.cram;
-
- int vmaj = cram_major_vers(in_c);
- int vmin = cram_minor_vers(in_c);
- if ((vers_maj != -1 && vers_maj != vmaj) ||
- (vers_min != -1 && vers_min != vmin)) {
- fprintf(samtools_stderr, "[%s] ERROR: input files have differing version numbers.\n",
- __func__);
+ if (firstfile->format.format != in->format.format) {
+ print_error("cat", "File %s is of different format!", fn[i]);
goto fail;
}
- vers_maj = vmaj;
- vers_min = vmin;
+ if (firstfile->format.format == cram) { //version check for cram
+ cram_fd *in_c;
+ in_c = in->fp.cram;
+
+ int vmaj = cram_major_vers(in_c);
+ int vmin = cram_minor_vers(in_c);
+ if ((vers_maj != -1 && vers_maj != vmaj) ||
+ (vers_min != -1 && vers_min != vmin)) {
+ fprintf(samtools_stderr, "[%s] ERROR: input files have differing version numbers.\n",
+ __func__);
+ goto fail;
+ }
+ vers_maj = vmaj;
+ vers_min = vmin;
+ }
old_h = sam_hdr_read(in);
if (!old_h) {
goto fail;
}
sam_hdr_destroy(old_h);
- sam_close(in);
+ old_h = NULL;
continue;
}
-
+ //merge RG lines
int old_count = sam_hdr_count_lines(old_h, "RG");
for (ki = 0; ki < old_count; ki++) {
const char *old_name = sam_hdr_line_name(old_h, "RG", ki);
}
}
- if (old_count > 1 && sam_hdr_count_lines(new_h, "RG") == old_count) {
+ if (firstfile->format.format == cram && old_count > 1 && sam_hdr_count_lines(new_h, "RG") == old_count) {
+ //RG order check for cram
for (ki = 0; ki < old_count; ki++) {
const char *old_name = sam_hdr_line_name(old_h, "RG", ki);
const char *new_name = sam_hdr_line_name(new_h, "RG", ki);
}
}
- sam_hdr_destroy(old_h);
- sam_close(in);
+ sam_hdr_destroy(old_h); old_h = NULL;
}
ks_free(&ks);
- *vers_maj_p = vers_maj;
- *vers_min_p = vers_min;
-
- return new_h;
+ if (vers_maj_p) {
+ *vers_maj_p = vers_maj;
+ }
+ if (vers_min_p) {
+ *vers_min_p = vers_min;
+ }
+ *out_h = new_h;
+ return files;
fail:
ks_free(&ks);
if (old_h) sam_hdr_destroy(old_h);
if (new_h) sam_hdr_destroy(new_h);
- if (in) sam_close(in);
+ *out_h = NULL;
+ for (i = 1; i < nfn; ++i) { //close files other than the firstfile
+ if (files[i]) {
+ sam_close(files[i]);
+ }
+ }
+ free(files);
return NULL;
}
-/*
- * CRAM files don't store the RG:Z:ID per read in the aux field.
- * Instead they have a numerical data series (RG) to point each read
- * back to the Nth @RG line in the file. This means that we may need
- * to edit the RG data series (if the files were produced from
- * "samtools split" for example).
- *
- * The encoding method is stored in the compression header. Typical
- * examples:
- *
- * RG => EXTERNAL {18} # Block content-id 18 holds RG values
- * # as a series of ITF8 encoded values
- *
- * RG => HUFFMAN {1, 255, 255, 255, 255, 255, 1, 0}
- * # One RG value #-1. (No RG)
- *
- * RG => HUFFMAN {1, 0, 1, 0} # One RG value #0 (always first RG)
- *
- * RG => HUFFMAN {2, 0, 1, 2, 1, 1}
- * # Two RG values, #0 and #1, written
- * # to the CORE block and possibly
- * # mixed with other data series.
- *
- * A single value can (but may not be) implemented as a zero bit
- * huffman code. In this situation we can change the meta-data in the
- * compression header to renumber an RG value..
+/* ----------------------------------------------------------------------
+ * CRAM cat
*/
-int cram_cat(int nfn, char * const *fn, const sam_hdr_t *h, const char* outcram, sam_global_args *ga, char *arg_list, int no_pg)
+
+// Reports the number of CRAM containers spanning a specified region if
+// specified, or the entire file if not.
+// This is the implements the "samtools cat -q [-r reg]" functionality.
+//
+// Returns 0 on success, <0 on error.
+static int cram_query_ncont(int nfn, char * const *fn, char *reg) {
+ int i;
+ hts_idx_t *idx = NULL;
+ sam_hdr_t *hdr = NULL;
+ hts_itr_t *iter = NULL;
+ samFile *in = NULL;
+
+ for (i = 0; i < nfn; i++) {
+ in = sam_open(fn[i], "r");
+ if (!in) {
+ print_error_errno("cat", "Couldn't open file %s", fn[i]);
+ return -1;
+ }
+ idx = sam_index_load(in, fn[i]);
+ if (!idx) {
+ print_error("cat", "No index found");
+ goto err;
+ }
+
+ off_t cstart = 0, cend = 0;
+ if (reg) {
+ sam_hdr_t *hdr = sam_hdr_read(in);
+ hts_itr_t *iter = sam_itr_querys(idx, hdr, reg);
+ if (!hdr) {
+ print_error("cat", "Unable to read header");
+ goto err;
+ }
+ if (!iter) {
+ print_error("cat", "Unable to parse region");
+ goto err;
+ }
+
+ if (cram_index_extents(in->fp.cram, iter->tid, iter->beg,
+ iter->end, &cstart, &cend) < 0) {
+ print_error("cat", "Failed to query index");
+ goto err;
+ }
+
+ hts_itr_destroy(iter);
+ sam_hdr_destroy(hdr);
+ }
+
+ int64_t first, last;
+ int64_t nc = cram_num_containers_between(in->fp.cram, cstart, cend,
+ &first, &last);
+ fprintf(samtools_stdout, "%s\t%"PRId64"\t%"PRId64"\t%"PRId64"\n",
+ fn[i], nc, first, last);
+ sam_close(in);
+ hts_idx_destroy(idx);
+ }
+
+ return 0;
+
+ err:
+ if (idx)
+ hts_idx_destroy(idx);
+ if (hdr)
+ sam_hdr_destroy(hdr);
+ if (iter)
+ hts_itr_destroy(iter);
+
+ sam_close(in);
+
+ return -1;
+}
+
+// Container range #:A-B or #:A.
+// Returns 0 on success, -1 on failure.
+static int cram_handle_cnum_region(cram_fd *fd, hts_idx_t *idx,
+ char *reg, off_t *cstart, off_t *cend) {
+ int cnum_start, cnum_end; // container versions
+ int n = sscanf(reg, "#:%d-%d", &cnum_start, &cnum_end);
+
+ if (n == 0) {
+ print_error("cat", "ERROR: Malformed region: %s", reg);
+ return -1;
+ } else if (n == 1) {
+ cnum_end = cnum_start;
+ }
+
+ int64_t nc = cram_num_containers(fd);
+ if (cnum_end >= nc) {
+ print_error("cat", "Too many containers. "
+ "The end range should be < %"PRId64, nc);
+ return -1;
+ }
+
+ // Container number to offset
+ *cstart = cram_container_num2offset(fd, cnum_start);
+ *cend = cram_container_num2offset(fd, cnum_end);
+ if (*cstart < 0 || *cend < 0)
+ return -1;
+
+ // Seek manually
+ return cram_seek(fd, *cstart, SEEK_SET);
+}
+
+// Normal range chr:start-end.
+// Returns an hts iterator on success, NULL on failure.
+static hts_itr_t *cram_handle_region(cram_fd *fd, hts_idx_t *idx, sam_hdr_t *h,
+ char *reg, off_t *cstart, off_t *cend) {
+ hts_itr_t *iter;
+
+ if (!idx) {
+ fprintf(samtools_stderr, "[%s] ERROR: No index found.\n", __func__);
+ return NULL;
+ }
+
+ // This does an implicit seek and modifies the cram_fd.
+ if (!(iter = sam_itr_querys(idx, h, reg))) {
+ print_error("cat", "Unable to parse region %s", reg);
+ return NULL;
+ }
+ if (cram_index_extents(fd, iter->tid, iter->beg,
+ iter->end, cstart, cend) < 0) {
+ print_error("cat", "Failed to query index");
+ return NULL;
+ }
+
+ return iter;
+}
+
+// Handle the -p A/B option to subdivide our region or file into portions.
+// Updates cstart/cend.
+// returns 0 on success (do something),
+// 1 on success (but nothing to do),
+// -1 on failure.
+static int cram_subdivide_part(cram_fd *fd, hts_idx_t *idx, char *part,
+ off_t *cstart, off_t *cend) {
+ int a, b;
+ // Part N of M
+ if (sscanf(part, "%d/%d", &a, &b) != 2) {
+ print_error("cat", "malformed region %s. Should be e.g. '1/10'", part);
+ return -1;
+ }
+
+ // Inclusive container numbers for range, 0 to NC-1
+ // Our part N/M is in container percentages as we can't have
+ // partial containers, so convert to that first.
+ int64_t cnum1, cnum2;
+ if (cstart) {
+ cnum1 = cram_container_offset2num(fd, *cstart);
+ cnum2 = cram_container_offset2num(fd, *cend);
+ } else {
+ cnum1 = 0;
+ cnum2 = cram_num_containers(fd)-1;
+ }
+
+ // Subdivide cnum1/cnum2 container numbers to new range cnum_start/end
+ int64_t nc = cnum2 - cnum1 + 1;
+ if (b > nc)
+ b = nc;
+
+ int cnum_start = (a-1)*(double)nc/b;
+ int cnum_end = a*(double)nc/b - 1;
+ if (cnum_start < 0 || cnum_end >= nc)
+ return 1;
+
+ // Then convert back to file offsets so we can seek and do htell
+ // to detect EOR/EOF.
+ *cstart = cram_container_num2offset(fd, cnum_start + cnum1);
+ *cend = cram_container_num2offset(fd, cnum_end + cnum1);
+
+ return 0;
+}
+
+// The main cram_cat interface.
+// Returns 0 on success, < 0 on error.
+int cram_cat(samFile * const firstfile, int nfn, char * const *fn,
+ const sam_hdr_t *h, const char* outcram, sam_global_args *ga,
+ char *arg_list, int no_pg, char *reg, char *part, int fast_reg)
{
- samFile *out;
+ samFile *out = NULL;
cram_fd *out_c;
- int i, vers_maj, vers_min;
+ int i, vers_maj, vers_min, ret = -1;
sam_hdr_t *new_h = NULL;
-
- /* Check consistent versioning and compatible headers */
- if (!(new_h = cram_cat_check_hdr(nfn, fn, h, &vers_maj, &vers_min)))
+ samFile **files = NULL;
+ hts_idx_t *idx = NULL;
+ hts_itr_t *iter = NULL;
+ sam_hdr_t *old_h = NULL;
+
+ // Check consistent versioning and compatible headers;
+ // merges RG lines, opens all files and returns them that multiple
+ // non-seekable stream inputs can be handled
+ if (!(files = cat_check_merge_hdr(firstfile, nfn, fn, h, &vers_maj,
+ &vers_min, &new_h)))
return -1;
+ if (!new_h) {
+ print_error_errno("cat", "failed to make output header");
+ goto closefiles;
+ }
+
/* Open the file with cram_vers */
char vers[100];
- sprintf(vers, "%d.%d", vers_maj, vers_min);
- out = sam_open_format(outcram, "wc", &ga->out);
+ snprintf(vers, sizeof(vers), "%d.%d", vers_maj, vers_min);
+
+ // Can override level=1 with e.g. "--output-fmt-option level=9"
+ out = sam_open_format(outcram, "wc1", &ga->out);
if (out == 0) {
print_error_errno("cat", "fail to open output file '%s'", outcram);
- return -1;
+ goto closefiles;
}
out_c = out->fp.cram;
cram_set_option(out_c, CRAM_OPT_VERSION, vers);
- //fprintf(samtools_stderr, "Creating cram vers %s\n", vers);
if (!no_pg && sam_hdr_add_pg(new_h, "samtools",
"VN", samtools_version(),
arg_list ? "CL": NULL,
arg_list ? arg_list : NULL,
NULL))
- return -1;
+ goto closefiles;
if (sam_hdr_write(out, new_h) < 0) {
print_error_errno("cat", "Couldn't write header");
- return -1;
+ goto closefiles;
}
+ out_c = out->fp.cram;
for (i = 0; i < nfn; ++i) {
samFile *in;
cram_fd *in_c;
cram_container *c;
- sam_hdr_t *old_h;
int new_rg = -1;
- in = sam_open(fn[i], "rc");
+ in = files[i];
if (in == 0) {
print_error_errno("cat", "fail to open file '%s'", fn[i]);
- return -1;
+ goto closefiles;
}
in_c = in->fp.cram;
old_h = sam_hdr_read(in);
if (!old_h) {
print_error("cat", "fail to read the header of file '%s'", fn[i]);
- return -1;
+ goto closefiles;
}
// Compute RG mapping if suitable for changing.
new_rg = sam_hdr_line_index(new_h, "RG", old_name);
if (new_rg < 0) {
print_error("cat", "fail to find @RG line '%s' in the new header", old_name);
- return -1;
+ goto closefiles;
}
} else {
print_error("cat", "fail to find @RG line in file '%s'", fn[i]);
- return -1;
+ goto closefiles;
}
} else {
new_rg = 0;
}
+ // We have multiple region syntax. Either the standard chr:start-end
+ // or a cat-specific #:num-num for explicit container numbers.
+ // Both of these seek to a specific file offset and also have the
+ // end offset known so we can use htell to detect when we're done.
+ // Those offsets are in cstart and cend.
+ //
+ // However for the e.g. -p 1/10 syntax we need to know the container
+ // numbers corresponding to the cstart/cend offsets as we can't
+ // start half way through a container when doing fractions.
+ // These are cnum1 and cnum2. (We could short cut this for the
+ // #:num-num, but it's simpler to just treat all regions identically.)
+ off_t cstart = 0, cend = 0;
+ int filter_by_cnum = 0;
+
+ if (reg || part) {
+ idx = sam_index_load(in, fn[i]);
+ if (!idx) {
+ print_error("cat", "failed to load index");
+ goto closefiles;
+ }
+ }
+
+ if (reg) {
+ if (strncmp(reg, "#:", 2) == 0) {
+ // Region as container numbers
+ if (cram_handle_cnum_region(in_c, idx, reg, &cstart,&cend) < 0)
+ goto closefiles;
+
+ filter_by_cnum = 1;
+ } else {
+ // Normal range chr:start-end
+ if (!(iter = cram_handle_region(in_c, idx, old_h, reg,
+ &cstart, &cend)))
+ goto closefiles;
+ }
+ }
+
+ // We can also take a range above and subdivide it into parts.
+ // Eg -r chr1 -p 1/10 (... to -p 10/10). Part only implies
+ // portions of the entire file.
+ if (part) {
+ int r = cram_subdivide_part(in_c, idx, part, &cstart, &cend);
+ if (r != 0) {
+ if (r > 0) // Not an error, just nothing to do
+ ret = 0;
+ goto closefiles;
+ }
+ }
+
+ if (cstart) // reg or part
+ if (0 != cram_seek(in_c, cstart, SEEK_SET))
+ goto closefiles;
+
+
+ // Make refid -2 ("*") come after other chromosomes, for easy sort
+ int itid = iter
+ ? (iter->tid == HTS_IDX_NOCOOR ? INT_MAX : iter->tid)
+ : 0;
+ int last_ref_id = -99;
+
+ off_t before_hdr = htell(cram_fd_get_fp(in_c));
+
// Copy contains and blocks within them
while ((c = cram_read_container(in_c))) {
if (cram_container_is_empty(in_c)) {
cram_block *blk;
// Container compression header
if (!(blk = cram_read_block(in_c)))
- return -1;
+ goto closefiles;
cram_free_block(blk);
cram_free_container(c);
continue;
}
+ int filter = 0;
+
// If we have just one RG key and new_rg != 0 then
// we need to edit the compression header. IF WE CAN.
if (new_rg) {
+ if (reg) {
+ print_error("cat", "Cannot specify a region while "
+ "transcoding RG lines");
+ goto closefiles;
+ }
int zero = 0;
//fprintf(samtools_stderr, "Transcode RG %d to %d\n", 0, new_rg);
cram_transcode_rg(in_c, out_c, c, 1, &zero, &new_rg);
} else {
int32_t num_slices;
- cram_block *blk;
- // Not switching rg so do the usual read/write loop
- if (cram_write_container(out_c, c) != 0)
- return -1;
+ if (reg) {
+ if (before_hdr > cend) {
+ cram_free_container(c);
+ break;
+ }
+ }
- // Container compression header
- if (!(blk = cram_read_block(in_c)))
- return -1;
- if (cram_write_block(out_c, blk) != 0) {
- cram_free_block(blk);
- return -1;
+ // For chr:start-end regions, do we need to filter or skip?
+ if (iter && reg) {
+ int refid;
+ hts_pos_t start, span, end;
+
+ cram_container_get_coords(c, &refid, &start, &span);
+ end = start+span;
+
+ // Make refid -1 ("*") come after other chromosomes
+ if (refid == -1)
+ refid = INT_MAX;
+
+ if (refid > itid || start > iter->end) {
+ // Beyond the requested range
+ break;
+ } else if (refid == -2) {
+ // Multi-ref containers. We only support this if
+ // the RI data series is in a container by itself.
+ filter = 3;
+ } else if (refid < itid || end < iter->beg) {
+ // Skip, in case of mixed size containers.
+ // Eg: Use "=", skip "-".
+ // ==========|======|=
+ // ---- ==|== ==|== ----
+ // ----- | ==== ----
+ filter = 2;
+ } else if (start < iter->beg || end > iter->end) {
+ // Container overlaps region.
+ // Fast mode just copies overlapping containers.
+ // Slow mode does a precise filtering by reading and
+ // filtering each record in turn so it's the same as
+ // a samtools view command.
+ filter = fast_reg ? 0 : 1;
+ }
+ // else we're in an "internal" container, so just copy
}
- cram_free_block(blk);
+ if (filter && last_ref_id == -1 && itid == INT_MAX)
+ // Multi-ref containers consisting solely of ref "*" are
+ // common, but if it's sorted then we know it's "*" from
+ // here on so we don't need to filter despite multi-ref.
+ filter = 0;
- // Container num_blocks can be invalid, due to a bug.
- // Instead we iterate in slice context instead.
- (void)cram_container_get_landmarks(c, &num_slices);
- cram_copy_slice(in_c, out_c, num_slices);
- }
+ if (filter) {
+ // Filter or skip
+ cram_filter_container(in_c, out_c, c, &last_ref_id);
+ } else {
+ // Copy. Consider adding a cram_copy_container API instead.
+ // Container compression header
+ cram_block *blk;
+ if (!(blk = cram_read_block(in_c)))
+ goto closefiles;
+
+ // Not switching rg so do the usual read/write loop
+ if (cram_write_container(out_c, c) != 0)
+ goto closefiles;
+
+ // Contatiner compression header
+ if (cram_write_block(out_c, blk) != 0) {
+ cram_free_block(blk);
+ goto closefiles;
+ }
+
+ // Container num_blocks can be invalid, due to a bug.
+ // Instead we iterate in slice context instead.
+ (void)cram_container_get_landmarks(c, &num_slices);
+ if (cram_copy_slice(in_c, out_c, num_slices) < 0) {
+ cram_free_block(blk);
+ goto closefiles;
+ }
+ cram_free_block(blk);
+ }
+ }
cram_free_container(c);
- }
+ // Location of next container start
+ before_hdr = htell(cram_fd_get_fp(in_c));
+ if (filter_by_cnum && before_hdr > cend)
+ break;
+ }
sam_hdr_destroy(old_h);
- sam_close(in);
+ old_h = NULL;
+
+ if (idx) {
+ hts_idx_destroy(idx);
+ idx = NULL;
+ }
+
+ if (iter) {
+ hts_itr_destroy(iter);
+ iter = NULL;
+ }
}
- sam_close(out);
- sam_hdr_destroy(new_h);
+ ret = 0;
- return 0;
-}
+closefiles:
+ if (old_h)
+ sam_hdr_destroy(old_h);
+
+ if (idx)
+ hts_idx_destroy(idx);
+
+ if (iter)
+ hts_itr_destroy(iter);
+ if (out)
+ sam_close(out);
+
+ if (new_h)
+ sam_hdr_destroy(new_h);
+
+ for (i = 1; i < nfn; ++i) { //skip firstfile and close rest
+ if (files[i]) {
+ sam_close(files[i]);
+ }
+ }
+ free(files);
+ return ret;
+}
+/* ----------------------------------------------------------------------
+ * BAM cat
+ */
#define BUF_SIZE 0x10000
#define GZIPID1 31
#define BGZF_EMPTY_BLOCK_SIZE 28
-int bam_cat(int nfn, char * const *fn, sam_hdr_t *h, const char* outbam, char *arg_list, int no_pg)
+int bam_cat(samFile * const firstfile, int nfn, char * const *fn, sam_hdr_t *h, const char* outbam, char *arg_list, int no_pg)
{
- BGZF *fp, *in = NULL;
+ BGZF *fp = NULL, *in = NULL;
uint8_t *buf = NULL;
uint8_t ebuf[BGZF_EMPTY_BLOCK_SIZE];
const int es=BGZF_EMPTY_BLOCK_SIZE;
int i;
+ samFile **files = NULL;
+ sam_hdr_t *new_h = NULL;
+ /* merges RG lines, opens all files and returns them that multiple non-seekable
+ stream inputs can be handled */
+ if (!(files = cat_check_merge_hdr(firstfile, nfn, fn, h, NULL, NULL, &new_h)))
+ return -1;
+ if (!new_h) {
+ print_error_errno("cat", "failed to make output header");
+ goto fail;
+ }
fp = strcmp(outbam, "-")? bgzf_open(outbam, "w") : bgzf_fdopen(fileno(samtools_stdout), "w");
if (fp == 0) {
print_error_errno("cat", "fail to open output file '%s'", outbam);
- return -1;
+ goto fail;
}
- if (h) {
- if (!no_pg && sam_hdr_add_pg(h, "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL))
- goto fail;
- if (bam_hdr_write(fp, h) < 0) {
- print_error_errno("cat", "Couldn't write header");
- goto fail;
- }
+ if (!no_pg && sam_hdr_add_pg(new_h, "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL))
+ goto fail;
+
+ if (bam_hdr_write(fp, new_h) < 0) {
+ print_error_errno("cat", "Couldn't write header");
+ goto fail;
}
buf = (uint8_t*) malloc(BUF_SIZE);
goto fail;
}
for(i = 0; i < nfn; ++i){
- sam_hdr_t *old;
int len,j;
-
- in = strcmp(fn[i], "-")? bgzf_open(fn[i], "r") : bgzf_fdopen(fileno(stdin), "r");
+ in = files[i]->fp.bgzf;
if (in == 0) {
print_error_errno("cat", "fail to open file '%s'", fn[i]);
goto fail;
}
- if (in->is_write) return -1;
-
- old = bam_hdr_read(in);
- if (old == NULL) {
- fprintf(samtools_stderr, "[%s] ERROR: couldn't read header for '%s'.\n",
- __func__, fn[i]);
- goto fail;
- }
- if (h == 0 && i == 0) {
- if (!no_pg && sam_hdr_add_pg(old, "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL))
- goto fail;
-
- if (bam_hdr_write(fp, old) < 0) {
- print_error_errno("cat", "Couldn't write header");
- goto fail;
- }
- }
+ if (in->is_write) goto fail;
if (in->block_offset < in->block_length) {
if (bgzf_write(fp, (char *)in->uncompressed_block + in->block_offset, in->block_length - in->block_offset) < 0) goto write_fail;
if (bgzf_raw_write(fp, ebuf, es) < 0) goto write_fail;
}
}
- sam_hdr_destroy(old);
- bgzf_close(in);
in = NULL;
}
free(buf);
if (bgzf_close(fp) < 0) {
fprintf(samtools_stderr, "[%s] Error on closing '%s'.\n", __func__, outbam);
- return -1;
+ goto fail;
+ }
+ for (i = 1; i < nfn; ++i) { //skip firstfile and close rest
+ if (files[i]) {
+ sam_close(files[i]);
+ }
}
+ free(files);
+ sam_hdr_destroy(new_h);
return 0;
write_fail:
fprintf(samtools_stderr, "[%s] Error writing to '%s'.\n", __func__, outbam);
fail:
- if (in) bgzf_close(in);
+ if (new_h) {
+ sam_hdr_destroy(new_h);
+ }
if (fp) bgzf_close(fp);
free(buf);
+
+ if (files) {
+ for(i = 1; i < nfn; ++i) { //except the firstfile
+ if(files[i]) {
+ sam_close(files[i]);
+ }
+ }
+ free(files);
+ }
return -1;
}
-
int main_cat(int argc, char *argv[])
{
sam_hdr_t *h = 0;
char *outfn = 0;
char **infns = NULL; // files to concatenate
int infns_size = 0;
- int c, ret = 0, no_pg = 0, usage = 0;
+ int c, ret = 0, no_pg = 0, usage = 0, query_ncont = 0, fast_mode = 0;
samFile *in;
sam_global_args ga;
+ char *reg = NULL, *part = NULL;
static const struct option lopts[] = {
- SAM_OPT_GLOBAL_OPTIONS('-', '-', '-', 0, '-', '@'),
+ SAM_OPT_GLOBAL_OPTIONS('-', '-', '-', 0, '-', '-'),
{"no-PG", no_argument, NULL, 1},
{ NULL, 0, NULL, 0 }
};
char *arg_list = NULL;
sam_global_args_init(&ga);
-
- while ((c = getopt_long(argc, argv, "h:o:b:@:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "h:o:b:r:p:qf", lopts, NULL)) >= 0) {
switch (c) {
case 'h': {
samFile *fph = sam_open(optarg, "r");
if (fph == 0) {
fprintf(samtools_stderr, "[%s] ERROR: fail to read the header from '%s'.\n", __func__, optarg);
- return 1;
+ ret = 1;
+ goto end;
}
h = sam_hdr_read(fph);
if (h == NULL) {
fprintf(samtools_stderr,
"[%s] ERROR: failed to read the header from '%s'.\n",
__func__, optarg);
- return 1;
+ ret = 1;
+ goto end;
}
sam_close(fph);
break;
case 1:
no_pg = 1;
break;
+ case 'r':
+ reg = optarg;
+ break;
+ case 'p':
+ part = optarg;
+ break;
+ case 'f':
+ fast_mode = 1;
+ break;
+ case 'q':
+ query_ncont=1;
+ break;
+
default:
if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
/* else fall-through */
if (!no_pg && !(arg_list = stringify_argv(argc+1, argv-1))) {
print_error("cat", "failed to create arg_list");
- return 1;
+ ret = 1;
+ goto end;
}
// Append files specified in argv to the list.
fprintf(samtools_stderr, " -h FILE copy the header from FILE [default is 1st input file]\n");
fprintf(samtools_stderr, " -o FILE output BAM/CRAM\n");
fprintf(samtools_stderr, " --no-PG do not add a PG line\n");
- sam_global_opt_help(samtools_stderr, "--..-@-.");
- return 1;
+ fprintf(samtools_stderr, "\nCRAM only options for filtering:\n");
+ fprintf(samtools_stderr, " -r REG filter to region REG.\n");
+ fprintf(samtools_stderr, " REG can also be #:cstart-cend for specific container numbers\n");
+ fprintf(samtools_stderr, " -p N/M Specify part N of M (where N is 1 to M inclusive)\n");
+ fprintf(samtools_stderr, " -f Fast mode: don't filter containers to exactly match region\n");
+ fprintf(samtools_stderr, " -q Query the total number of indexed containers\n");
+ fprintf(samtools_stderr, "\nStandard options:\n");
+ sam_global_opt_help(samtools_stderr, "---.---.");
+ ret = 1;
+ goto end;
}
in = sam_open(infns[0], "r");
if (!in) {
print_error_errno("cat", "failed to open file '%s'", infns[0]);
- return 1;
+ ret = 1;
+ goto end;
}
switch (hts_get_format(in)->format) {
case bam:
- sam_close(in);
- if (bam_cat(infns_size+nargv_fns, infns, h, outfn? outfn : "-", arg_list, no_pg) < 0)
+ if (bam_cat(in, infns_size+nargv_fns, infns, h, outfn? outfn : "-", arg_list, no_pg) < 0)
ret = 1;
break;
case cram:
- sam_close(in);
- if (cram_cat(infns_size+nargv_fns, infns, h, outfn? outfn : "-", &ga, arg_list, no_pg) < 0)
+ if (query_ncont) {
+ if (cram_query_ncont(infns_size+nargv_fns, infns, reg) < 0)
+ ret = 1;
+ } else {
+ if (cram_cat(in, infns_size+nargv_fns, infns, h,
+ outfn? outfn : "-", &ga, arg_list, no_pg, reg,
+ part, fast_mode) < 0)
ret = 1;
+ }
break;
default:
- sam_close(in);
fprintf(samtools_stderr, "[%s] ERROR: input is not BAM or CRAM\n", __func__);
- return 1;
+ ret = 1;
}
+ sam_close(in);
end:
if (infns_size > 0) {
free(arg_list);
if (h)
sam_hdr_destroy(h);
+ sam_global_args_free(&ga);
return ret;
}
/* bam_consensus.c -- consensus subcommand.
Copyright (C) 1998-2001,2003 Medical Research Council (Gap4/5 source)
- Copyright (C) 2003-2005,2007-2023 Genome Research Ltd.
+ Copyright (C) 2003-2005,2007-2024 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
// Ignore ambiguous bases in seq for now, so we don't treat R, Y, etc
// as part of one base and part another. Based on BAM seqi values.
// We also use freq[16] as "*" for gap.
- int freq[17] = {0}; // base frequency, aka depth
- int score[17] = {0}; // summation of base qualities
+ int freq[17] = {0}; // base frequency, aka depth
+ uint64_t score[17] = {0}; // summation of base qualities
// Accumulate
for (; plp; plp = plp->next) {
}
// Total usable depth
- int tscore = 0;
+ uint64_t tscore = 0;
for (i = 0; i < 5; i++)
tscore += score[1<<i];
// Best and second best potential calls
- int call1 = 15, call2 = 15;
- int score1 = 0, score2 = 0;
+ int call1 = 15, call2 = 15;
+ uint64_t score1 = 0, score2 = 0;
for (i = 0; i < 5; i++) {
int c = 1<<i; // A C G T *
if (score1 < score[c]) {
}
// Work out which best and second best are usable as a call
- int used_score = score1;
- int used_base = call1;
+ uint64_t used_score = score1;
+ int used_base = call1;
if (score2 >= opts->het_fract * score1 && opts->ambig) {
used_base |= call2;
used_score += score2;
calculate_consensus_gap5m(pos, opts->use_mqual ? CONS_MQUAL : 0,
depth, p, opts, &cons, opts->default_qual,
&cons_prob_recall, &cons_prob_precise);
- if (cons.het_logodd > 0 && opts->ambig) {
+ if (cons.depth < opts->min_depth) {
+ cb = 'N';
+ cq = 0;
+ } else if (cons.het_logodd > 0 && opts->ambig) {
cb = "AMRWa" // 5x5 matrix with ACGT* per row / col
"MCSYc"
"RSGKg"
calculate_consensus_gap5m(pos, opts->use_mqual ? CONS_MQUAL : 0,
depth, p, opts, &cons, opts->default_qual,
&cons_prob_recall, &cons_prob_precise);
- if (cons.het_logodd > 0 && opts->ambig) {
+ if (cons.depth < opts->min_depth) {
+ cb = 'N';
+ cq = 0;
+ } else if (cons.het_logodd > 0 && opts->ambig) {
cb = "AMRWa" // 5x5 matrix with ACGT* per row / col
"MCSYc"
"RSGKg"
"WYKTt"
"acgt*"[cons.het_call];
cq = cons.het_logodd;
- } else{
+ } else {
cb = "ACGT*"[cons.call];
cq = cons.phred;
}
fprintf(fp, " --show-ins yes/no Whether to show insertions [yes]\n");
fprintf(fp, " --mark-ins Add '+' before every inserted base/qual [off]\n");
fprintf(fp, " -A, --ambig Enable IUPAC ambiguity codes [off]\n");
+ fprintf(fp, " -d, --min-depth INT Minimum depth of INT [1]\n");
fprintf(fp, "\nFor simple consensus mode:\n");
fprintf(fp, " -q, --(no-)use-qual Use quality values in calculation [off]\n");
fprintf(fp, " -c, --call-fract INT At least INT portion of bases must agree [0.75]\n");
- fprintf(fp, " -d, --min-depth INT Minimum depth of INT [2]\n");
fprintf(fp, " -H, --het-fract INT Minimum fraction of 2nd-most to most common base [0.15]\n");
fprintf(fp, "\nFor default \"Bayesian\" consensus mode:\n");
fprintf(fp, " -C, --cutoff C Consensus cutoff quality C [10]\n");
/* bam_consensus.c -- consensus subcommand.
Copyright (C) 1998-2001,2003 Medical Research Council (Gap4/5 source)
- Copyright (C) 2003-2005,2007-2023 Genome Research Ltd.
+ Copyright (C) 2003-2005,2007-2024 Genome Research Ltd.
Author: James Bonfield <jkb@sanger.ac.uk>
// Ignore ambiguous bases in seq for now, so we don't treat R, Y, etc
// as part of one base and part another. Based on BAM seqi values.
// We also use freq[16] as "*" for gap.
- int freq[17] = {0}; // base frequency, aka depth
- int score[17] = {0}; // summation of base qualities
+ int freq[17] = {0}; // base frequency, aka depth
+ uint64_t score[17] = {0}; // summation of base qualities
// Accumulate
for (; plp; plp = plp->next) {
}
// Total usable depth
- int tscore = 0;
+ uint64_t tscore = 0;
for (i = 0; i < 5; i++)
tscore += score[1<<i];
// Best and second best potential calls
- int call1 = 15, call2 = 15;
- int score1 = 0, score2 = 0;
+ int call1 = 15, call2 = 15;
+ uint64_t score1 = 0, score2 = 0;
for (i = 0; i < 5; i++) {
int c = 1<<i; // A C G T *
if (score1 < score[c]) {
}
// Work out which best and second best are usable as a call
- int used_score = score1;
- int used_base = call1;
+ uint64_t used_score = score1;
+ int used_base = call1;
if (score2 >= opts->het_fract * score1 && opts->ambig) {
used_base |= call2;
used_score += score2;
calculate_consensus_gap5m(pos, opts->use_mqual ? CONS_MQUAL : 0,
depth, p, opts, &cons, opts->default_qual,
&cons_prob_recall, &cons_prob_precise);
- if (cons.het_logodd > 0 && opts->ambig) {
+ if (cons.depth < opts->min_depth) {
+ cb = 'N';
+ cq = 0;
+ } else if (cons.het_logodd > 0 && opts->ambig) {
cb = "AMRWa" // 5x5 matrix with ACGT* per row / col
"MCSYc"
"RSGKg"
calculate_consensus_gap5m(pos, opts->use_mqual ? CONS_MQUAL : 0,
depth, p, opts, &cons, opts->default_qual,
&cons_prob_recall, &cons_prob_precise);
- if (cons.het_logodd > 0 && opts->ambig) {
+ if (cons.depth < opts->min_depth) {
+ cb = 'N';
+ cq = 0;
+ } else if (cons.het_logodd > 0 && opts->ambig) {
cb = "AMRWa" // 5x5 matrix with ACGT* per row / col
"MCSYc"
"RSGKg"
"WYKTt"
"acgt*"[cons.het_call];
cq = cons.het_logodd;
- } else{
+ } else {
cb = "ACGT*"[cons.call];
cq = cons.phred;
}
fprintf(fp, " --show-ins yes/no Whether to show insertions [yes]\n");
fprintf(fp, " --mark-ins Add '+' before every inserted base/qual [off]\n");
fprintf(fp, " -A, --ambig Enable IUPAC ambiguity codes [off]\n");
+ fprintf(fp, " -d, --min-depth INT Minimum depth of INT [1]\n");
fprintf(fp, "\nFor simple consensus mode:\n");
fprintf(fp, " -q, --(no-)use-qual Use quality values in calculation [off]\n");
fprintf(fp, " -c, --call-fract INT At least INT portion of bases must agree [0.75]\n");
- fprintf(fp, " -d, --min-depth INT Minimum depth of INT [2]\n");
fprintf(fp, " -H, --het-fract INT Minimum fraction of 2nd-most to most common base [0.15]\n");
fprintf(fp, "\nFor default \"Bayesian\" consensus mode:\n");
fprintf(fp, " -C, --cutoff C Consensus cutoff quality C [10]\n");
/* bam_fastq.c -- FASTA and FASTQ file generation
- Copyright (C) 2009-2017, 2019-2020, 2023 Genome Research Ltd.
+ Copyright (C) 2009-2017, 2019-2020, 2023-2024 Genome Research Ltd.
Portions copyright (C) 2009, 2011, 2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
#include "htslib/kstring.h"
#include "htslib/bgzf.h"
#include "htslib/thread_pool.h"
+#include "htslib/khash.h"
#include "samtools.h"
#include "sam_opts.h"
+KHASH_SET_INIT_STR(str)
+typedef khash_t(str) strhash_t;
+
#define DEFAULT_BARCODE_TAG "BC"
#define DEFAULT_QUALITY_TAG "QT"
#define INDEX_SEPARATOR "+"
" paired reads will be written to the -1 and -2 files.\n"
" -d, --tag TAG[:VAL]\n"
" only include reads containing TAG, optionally with value VAL\n"
+" -D, --tag-file STR:FILE\n"
+" only include reads containing TAG, with a value listed in FILE\n"
" -f, --require-flags INT\n"
" only include reads with all of the FLAGs in INT present [0]\n" // F&x == x
" -F, --excl[ude]-flags INT\n"
char *extra_tags;
char compression_level;
const char *filter_tag; // -d opt
- const char *filter_value_str;
- int64_t filter_value_int;
- float filter_value_flt;
+ strhash_t *filter_tag_vals;
} bam2fq_opts_t;
typedef struct bam2fq_state {
htsThreadPool p;
} bam2fq_state_t;
+// Adds a single tag value to the filter tag value hash
+static int add_tag_value(bam2fq_opts_t *opts, char *val) {
+ if (!opts->filter_tag_vals) {
+ if (!(opts->filter_tag_vals = kh_init(str)))
+ return -1;
+ }
+
+ if (!(val = strdup(val)))
+ return -1;
+
+ int ret = 0;
+ kh_put(str, opts->filter_tag_vals, val, &ret);
+ if (ret <= 0)
+ free(val);
+
+ return ret < 0 ? -1 : 0;
+}
+
+// Adds multiple values, listed in a file
+static int add_tag_file(bam2fq_opts_t *opts, char *fn) {
+ FILE *fp;
+ kstring_t ks = {0,0};
+
+ if (!(fp = fopen(fn, "r"))) {
+ print_error_errno("fastq", "failed to open \"%s\" for reading", fn);
+ return -1;
+ }
+
+ while (ks.l = 0, kgetline(&ks, (kgets_func *)fgets, fp) >= 0) {
+ if (add_tag_value(opts, ks.s) < 0) {
+ ks_free(&ks);
+ return -1;
+ }
+ }
+
+ ks_free(&ks);
+ return 0;
+}
+
static readpart which_readpart(const bam1_t *b)
{
if ((b->core.flag & BAM_FREAD1) && !(b->core.flag & BAM_FREAD2)) {
static void free_opts(bam2fq_opts_t *opts)
{
+ if (opts->filter_tag_vals) {
+ khint_t k;
+ for (k = 0; k < kh_end(opts->filter_tag_vals); k++)
+ if (kh_exist(opts->filter_tag_vals, k))
+ free((char *)kh_key(opts->filter_tag_vals, k));
+
+ kh_destroy(str, opts->filter_tag_vals);
+ }
free(opts);
}
{"barcode-tag", required_argument, NULL, 'b'},
{"quality-tag", required_argument, NULL, 'q'},
{"tag", required_argument, NULL, 'd'},
+ {"tag-file", required_argument, NULL, 'D'},
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "0:1:2:o:f:F:G:niNOs:c:tT:v:@:d:",
+ while ((c = getopt_long(argc, argv, "0:1:2:o:f:F:G:niNOs:c:tT:v:@:d:D:",
lopts, NULL)) > 0) {
switch (c) {
case 'b': opts->barcode_tag = optarg; break;
return false;
}
+ if (opts->filter_tag && memcmp(opts->filter_tag, optarg, 2)) {
+ print_error("fastq", "Different tag type specified "
+ "to before");
+ free_opts(opts);
+ return false;
+ }
+
+ if (strlen(optarg) >= 3)
+ add_tag_value(opts, optarg+3);
+ opts->filter_tag = optarg;
+ break;
+
+ case 'D':
+ // Allow ";" as delimiter besides ":" to support MinGW CLI POSIX
+ // path translation as described at:
+ // http://www.mingw.org/wiki/Posix_path_conversion
+ if (strlen(optarg) < 4
+ || (optarg[2] != ':' && optarg[2] != ';')) {
+ print_error("view", "Invalid \"tag:file\" option: \"%s\"",
+ optarg);
+ free_opts(opts);
+ return false;
+ }
+
+ if (opts->filter_tag && memcmp(opts->filter_tag, optarg, 2)) {
+ print_error("fastq", "Different tag type specified "
+ "to before");
+ free_opts(opts);
+ return false;
+ }
+
+ if (strlen(optarg) >= 3) {
+ if (add_tag_file(opts, optarg+3) < 0) {
+ free_opts(opts);
+ return false;
+ }
+ }
opts->filter_tag = optarg;
- opts->filter_value_str = strlen(optarg) > 2 ? optarg+3 : NULL;
- opts->filter_value_int = INT64_MAX; // fill out later
- opts->filter_value_flt = FLT_MAX;
break;
case '?':
if (!s)
return true;
- if (opts->filter_value_str) {
+ if (opts->filter_tag_vals) {
+ char t[32], *val = t;
switch (*s) {
case 'i': case 'I':
case 's': case 'S':
case 'c': case 'C':
- if (opts->filter_value_int == INT64_MAX)
- // cache integer conversion for repeated use
- opts->filter_value_int =
- strtoll(opts->filter_value_str, NULL, 0);
- if (opts->filter_value_int != bam_aux2i(s))
+ if (snprintf(t, 32, "%"PRId64, bam_aux2i(s)) <= 0)
return true;
break;
case 'f':
- if (opts->filter_value_flt == FLT_MAX)
- opts->filter_value_flt = atof(opts->filter_value_str);
// Comparing floats is hard.
// Eg (double)0.1 - (double)0.1f is -1.5e-9.
// Given BAM binary encoding is float however, just keep it.
// This means rounding errors will (hopefully) always be the
// same and basic equality still works.
- if (opts->filter_value_flt != (float)bam_aux2f(s))
+ if (snprintf(t, 32, "%f", (float)bam_aux2f(s)) <= 0)
return true;
break;
case 'A':
- if (s[1] != *opts->filter_value_str)
- return true;
+ t[0] = s[1];
+ t[1] = 0;
break;
case 'Z': case 'H':
- if (strcmp((char *)s+1, opts->filter_value_str) != 0)
- return true;
+ val = (char *)s+1;
break;
default:
// Anything unsupported fails the filter match too.
return true;
}
+
+ khint_t k = kh_get(str, opts->filter_tag_vals, val);
+ if (k == kh_end(opts->filter_tag_vals))
+ return 1; // tag value not found
}
}
}
char *bc_end = bc, *qt_end = qt;
- while (len ? *bc_end && rem-- : isalpha(*bc_end))
- bc_end++, qt_end += qt != NULL;
+ if (qt) {
+ while (len ? *bc_end && rem-- : isalpha(*bc_end))
+ bc_end++, qt_end++;
+ } else {
+ while (len ? *bc_end && rem-- : isalpha(*bc_end))
+ bc_end++;
+ }
switch (fc) {
case 'n':
/* bam_fastq.c -- FASTA and FASTQ file generation
- Copyright (C) 2009-2017, 2019-2020, 2023 Genome Research Ltd.
+ Copyright (C) 2009-2017, 2019-2020, 2023-2024 Genome Research Ltd.
Portions copyright (C) 2009, 2011, 2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
#include "htslib/kstring.h"
#include "htslib/bgzf.h"
#include "htslib/thread_pool.h"
+#include "htslib/khash.h"
#include "samtools.h"
#include "sam_opts.h"
+KHASH_SET_INIT_STR(str)
+typedef khash_t(str) strhash_t;
+
#define DEFAULT_BARCODE_TAG "BC"
#define DEFAULT_QUALITY_TAG "QT"
#define INDEX_SEPARATOR "+"
" paired reads will be written to the -1 and -2 files.\n"
" -d, --tag TAG[:VAL]\n"
" only include reads containing TAG, optionally with value VAL\n"
+" -D, --tag-file STR:FILE\n"
+" only include reads containing TAG, with a value listed in FILE\n"
" -f, --require-flags INT\n"
" only include reads with all of the FLAGs in INT present [0]\n" // F&x == x
" -F, --excl[ude]-flags INT\n"
char *extra_tags;
char compression_level;
const char *filter_tag; // -d opt
- const char *filter_value_str;
- int64_t filter_value_int;
- float filter_value_flt;
+ strhash_t *filter_tag_vals;
} bam2fq_opts_t;
typedef struct bam2fq_state {
htsThreadPool p;
} bam2fq_state_t;
+// Adds a single tag value to the filter tag value hash
+static int add_tag_value(bam2fq_opts_t *opts, char *val) {
+ if (!opts->filter_tag_vals) {
+ if (!(opts->filter_tag_vals = kh_init(str)))
+ return -1;
+ }
+
+ if (!(val = strdup(val)))
+ return -1;
+
+ int ret = 0;
+ kh_put(str, opts->filter_tag_vals, val, &ret);
+ if (ret <= 0)
+ free(val);
+
+ return ret < 0 ? -1 : 0;
+}
+
+// Adds multiple values, listed in a file
+static int add_tag_file(bam2fq_opts_t *opts, char *fn) {
+ FILE *fp;
+ kstring_t ks = {0,0};
+
+ if (!(fp = fopen(fn, "r"))) {
+ print_error_errno("fastq", "failed to open \"%s\" for reading", fn);
+ return -1;
+ }
+
+ while (ks.l = 0, kgetline(&ks, (kgets_func *)fgets, fp) >= 0) {
+ if (add_tag_value(opts, ks.s) < 0) {
+ ks_free(&ks);
+ return -1;
+ }
+ }
+
+ ks_free(&ks);
+ return 0;
+}
+
static readpart which_readpart(const bam1_t *b)
{
if ((b->core.flag & BAM_FREAD1) && !(b->core.flag & BAM_FREAD2)) {
static void free_opts(bam2fq_opts_t *opts)
{
+ if (opts->filter_tag_vals) {
+ khint_t k;
+ for (k = 0; k < kh_end(opts->filter_tag_vals); k++)
+ if (kh_exist(opts->filter_tag_vals, k))
+ free((char *)kh_key(opts->filter_tag_vals, k));
+
+ kh_destroy(str, opts->filter_tag_vals);
+ }
free(opts);
}
{"barcode-tag", required_argument, NULL, 'b'},
{"quality-tag", required_argument, NULL, 'q'},
{"tag", required_argument, NULL, 'd'},
+ {"tag-file", required_argument, NULL, 'D'},
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "0:1:2:o:f:F:G:niNOs:c:tT:v:@:d:",
+ while ((c = getopt_long(argc, argv, "0:1:2:o:f:F:G:niNOs:c:tT:v:@:d:D:",
lopts, NULL)) > 0) {
switch (c) {
case 'b': opts->barcode_tag = optarg; break;
return false;
}
+ if (opts->filter_tag && memcmp(opts->filter_tag, optarg, 2)) {
+ print_error("fastq", "Different tag type specified "
+ "to before");
+ free_opts(opts);
+ return false;
+ }
+
+ if (strlen(optarg) >= 3)
+ add_tag_value(opts, optarg+3);
+ opts->filter_tag = optarg;
+ break;
+
+ case 'D':
+ // Allow ";" as delimiter besides ":" to support MinGW CLI POSIX
+ // path translation as described at:
+ // http://www.mingw.org/wiki/Posix_path_conversion
+ if (strlen(optarg) < 4
+ || (optarg[2] != ':' && optarg[2] != ';')) {
+ print_error("view", "Invalid \"tag:file\" option: \"%s\"",
+ optarg);
+ free_opts(opts);
+ return false;
+ }
+
+ if (opts->filter_tag && memcmp(opts->filter_tag, optarg, 2)) {
+ print_error("fastq", "Different tag type specified "
+ "to before");
+ free_opts(opts);
+ return false;
+ }
+
+ if (strlen(optarg) >= 3) {
+ if (add_tag_file(opts, optarg+3) < 0) {
+ free_opts(opts);
+ return false;
+ }
+ }
opts->filter_tag = optarg;
- opts->filter_value_str = strlen(optarg) > 2 ? optarg+3 : NULL;
- opts->filter_value_int = INT64_MAX; // fill out later
- opts->filter_value_flt = FLT_MAX;
break;
case '?':
if (!s)
return true;
- if (opts->filter_value_str) {
+ if (opts->filter_tag_vals) {
+ char t[32], *val = t;
switch (*s) {
case 'i': case 'I':
case 's': case 'S':
case 'c': case 'C':
- if (opts->filter_value_int == INT64_MAX)
- // cache integer conversion for repeated use
- opts->filter_value_int =
- strtoll(opts->filter_value_str, NULL, 0);
- if (opts->filter_value_int != bam_aux2i(s))
+ if (snprintf(t, 32, "%"PRId64, bam_aux2i(s)) <= 0)
return true;
break;
case 'f':
- if (opts->filter_value_flt == FLT_MAX)
- opts->filter_value_flt = atof(opts->filter_value_str);
// Comparing floats is hard.
// Eg (double)0.1 - (double)0.1f is -1.5e-9.
// Given BAM binary encoding is float however, just keep it.
// This means rounding errors will (hopefully) always be the
// same and basic equality still works.
- if (opts->filter_value_flt != (float)bam_aux2f(s))
+ if (snprintf(t, 32, "%f", (float)bam_aux2f(s)) <= 0)
return true;
break;
case 'A':
- if (s[1] != *opts->filter_value_str)
- return true;
+ t[0] = s[1];
+ t[1] = 0;
break;
case 'Z': case 'H':
- if (strcmp((char *)s+1, opts->filter_value_str) != 0)
- return true;
+ val = (char *)s+1;
break;
default:
// Anything unsupported fails the filter match too.
return true;
}
+
+ khint_t k = kh_get(str, opts->filter_tag_vals, val);
+ if (k == kh_end(opts->filter_tag_vals))
+ return 1; // tag value not found
}
}
}
char *bc_end = bc, *qt_end = qt;
- while (len ? *bc_end && rem-- : isalpha(*bc_end))
- bc_end++, qt_end += qt != NULL;
+ if (qt) {
+ while (len ? *bc_end && rem-- : isalpha(*bc_end))
+ bc_end++, qt_end++;
+ } else {
+ while (len ? *bc_end && rem-- : isalpha(*bc_end))
+ bc_end++;
+ }
switch (fc) {
case 'n':
* samtools import a_1.fq a_2.fq
* samtools import a_interleaved.fq
*
- * Copyright (C) 2020-2021, 2023 Genome Research Ltd.
+ * Copyright (C) 2020-2021, 2023-2024 Genome Research Ltd.
*
* Author: James Bonfield <jkb@sanger.ac.uk>
*/
fprintf(fp, " -u Uncompressed output\n");
fprintf(fp, " --order TAG Store Nth record count in TAG\n");
fprintf(fp, "\n");
+ fprintf(fp, " --no-PG Do not add a PG line\n");
sam_global_opt_help(fp, "-.O.-@--");
fprintf(fp, "\nA single fastq file will be interpreted as -s, -0 or -1 depending on\n");
if (argc == 1)
opts->fn[FQ_SINGLE] = argv[0];
else
- for (i = 0; i < 4; i++)
+ for (i = 0; i < 2; i++)
if (argc > i)
opts->fn[FQ_R1+i] = argv[i];
goto err;
}
+ if (!opts->no_pg) {
+ char *arg_list;
+ if (!(arg_list = stringify_argv(argc+1+optind, argv-1-optind))) {
+ print_error("view", "failed to create arg_list");
+ goto err;
+ }
+ if (sam_hdr_add_pg(hdr_out, "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL" : NULL,
+ arg_list ? arg_list : NULL,
+ NULL)) {
+ fprintf(stderr, "Failed to add PG line to the header");
+ free(arg_list);
+ goto err;
+ }
+
+ free(arg_list);
+ }
+
// Read group
if (opts->rg_line) {
if (*opts->rg_line != '@')
// Interleave / combine from n files (ids[0..n-1]).
- int res;
+ int res = 0;
int eof = 0;
do {
idx_seq.l = idx_qual.l = 0;
* samtools import a_1.fq a_2.fq
* samtools import a_interleaved.fq
*
- * Copyright (C) 2020-2021, 2023 Genome Research Ltd.
+ * Copyright (C) 2020-2021, 2023-2024 Genome Research Ltd.
*
* Author: James Bonfield <jkb@sanger.ac.uk>
*/
fprintf(fp, " -u Uncompressed output\n");
fprintf(fp, " --order TAG Store Nth record count in TAG\n");
fprintf(fp, "\n");
+ fprintf(fp, " --no-PG Do not add a PG line\n");
sam_global_opt_help(fp, "-.O.-@--");
fprintf(fp, "\nA single fastq file will be interpreted as -s, -0 or -1 depending on\n");
if (argc == 1)
opts->fn[FQ_SINGLE] = argv[0];
else
- for (i = 0; i < 4; i++)
+ for (i = 0; i < 2; i++)
if (argc > i)
opts->fn[FQ_R1+i] = argv[i];
goto err;
}
+ if (!opts->no_pg) {
+ char *arg_list;
+ if (!(arg_list = stringify_argv(argc+1+optind, argv-1-optind))) {
+ print_error("view", "failed to create arg_list");
+ goto err;
+ }
+ if (sam_hdr_add_pg(hdr_out, "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL" : NULL,
+ arg_list ? arg_list : NULL,
+ NULL)) {
+ fprintf(samtools_stderr, "Failed to add PG line to the header");
+ free(arg_list);
+ goto err;
+ }
+
+ free(arg_list);
+ }
+
// Read group
if (opts->rg_line) {
if (*opts->rg_line != '@')
// Interleave / combine from n files (ids[0..n-1]).
- int res;
+ int res = 0;
int eof = 0;
do {
idx_seq.l = idx_qual.l = 0;
/* bam_index.c -- index and idxstats subcommands.
- Copyright (C) 2008-2011, 2013-2016, 2018, 2019, 2023 Genome Research Ltd.
+ Copyright (C) 2008-2011, 2013-2016, 2018, 2019, 2023-2024 Genome Research Ltd.
Portions copyright (C) 2010 Broad Institute.
Portions copyright (C) 2013 Peter Cock, The James Hutton Institute.
static void usage_exit(FILE *fp, int exit_status)
{
- fprintf(fp, "Usage: samtools idxstats [options] <in.bam>\n");
+ fprintf(fp, "Usage: samtools idxstats [options] <in.bam>\n"
+ " -X Include customized index file\n");
sam_global_opt_help(fp, "-.---@-.");
exit(exit_status);
}
hts_idx_t* idx;
sam_hdr_t* header;
samFile* fp;
- int c;
+ int c, has_index_file = 0, file_names = 1;
+ char *index_name = NULL;
sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
static const struct option lopts[] = {
{NULL, 0, NULL, 0}
};
- while ((c = getopt_long(argc, argv, "@:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "@:X", lopts, NULL)) >= 0) {
switch (c) {
- default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
- /* else fall-through */
- case '?':
- usage_exit(stderr, EXIT_FAILURE);
+ case 'X': has_index_file=1; break;
+ default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
+ /* else fall-through */
+ case '?':
+ usage_exit(stderr, EXIT_FAILURE);
}
}
- if (argc != optind+1) {
+ if (has_index_file) {
+ file_names = 2;
+ index_name = argv[optind + 1];
+ }
+
+ if (argc != optind + file_names) {
if (argc == optind) usage_exit(stdout, EXIT_SUCCESS);
else usage_exit(stderr, EXIT_FAILURE);
}
return 1;
}
} else {
- idx = sam_index_load(fp, argv[optind]);
+ idx = sam_index_load2(fp, argv[optind], index_name);
if (idx == NULL) {
print_error("idxstats", "fail to load index for \"%s\", "
"reverting to slow method", argv[optind]);
/* bam_index.c -- index and idxstats subcommands.
- Copyright (C) 2008-2011, 2013-2016, 2018, 2019, 2023 Genome Research Ltd.
+ Copyright (C) 2008-2011, 2013-2016, 2018, 2019, 2023-2024 Genome Research Ltd.
Portions copyright (C) 2010 Broad Institute.
Portions copyright (C) 2013 Peter Cock, The James Hutton Institute.
static void usage_exit(FILE *fp, int exit_status)
{
- fprintf(fp, "Usage: samtools idxstats [options] <in.bam>\n");
+ fprintf(fp, "Usage: samtools idxstats [options] <in.bam>\n"
+ " -X Include customized index file\n");
sam_global_opt_help(fp, "-.---@-.");
samtools_exit(exit_status);
}
hts_idx_t* idx;
sam_hdr_t* header;
samFile* fp;
- int c;
+ int c, has_index_file = 0, file_names = 1;
+ char *index_name = NULL;
sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
static const struct option lopts[] = {
{NULL, 0, NULL, 0}
};
- while ((c = getopt_long(argc, argv, "@:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "@:X", lopts, NULL)) >= 0) {
switch (c) {
- default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
- /* else fall-through */
- case '?':
- usage_exit(samtools_stderr, EXIT_FAILURE);
+ case 'X': has_index_file=1; break;
+ default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
+ /* else fall-through */
+ case '?':
+ usage_exit(samtools_stderr, EXIT_FAILURE);
}
}
- if (argc != optind+1) {
+ if (has_index_file) {
+ file_names = 2;
+ index_name = argv[optind + 1];
+ }
+
+ if (argc != optind + file_names) {
if (argc == optind) usage_exit(samtools_stdout, EXIT_SUCCESS);
else usage_exit(samtools_stderr, EXIT_FAILURE);
}
return 1;
}
} else {
- idx = sam_index_load(fp, argv[optind]);
+ idx = sam_index_load2(fp, argv[optind], index_name);
if (idx == NULL) {
print_error("idxstats", "fail to load index for \"%s\", "
"reverting to slow method", argv[optind]);
int opt;
int beg;
int end;
+ int len;
} check_t;
typedef struct {
return ret;
}
- if (strncmp(original + o_beg, duplicate + d_beg, o_end - o_beg) == 0) {
+ int o_len = o_end - o_beg;
+ int d_len = d_end - d_beg;
+
+ if ((o_len == d_len) && memcmp(original + o_beg, duplicate + d_beg, o_len) == 0) {
long xdiff, ydiff;
if (ox > dx) {
return ret;
}
- if (strncmp(name + o_beg, duplicate + d_beg, o_end - o_beg) == 0) {
+ int o_len = o_end - o_beg;
+ int d_len = d_end - d_beg;
+
+ if ((o_len == d_len) && memcmp(name + o_beg, duplicate + d_beg, o_len) == 0) {
// the initial parts match, look at the numbers
long xdiff, ydiff;
c->y = dy;
c->beg = d_beg;
c->end = d_end;
+ c->len = d_end - d_beg;
return ret;
}
}
-static int xcoord_sort(const void *a, const void *b) {
+static int chain_sort(const void *a, const void *b) {
check_t *ac = (check_t *) a;
check_t *bc = (check_t *) b;
+ int ret;
+
+ if ((ret = ac->len - bc->len))
+ return ret;
+ else if ((ret = memcmp(bam_get_qname(ac->b) + ac->beg, bam_get_qname(bc->b) + bc->beg, ac->len)))
+ return ret;
return (ac->x - bc->x);
}
int ret = 0;
size_t curr = 0;
- qsort(list->c, list->length, sizeof(list->c[0]), xcoord_sort);
+ qsort(list->c, list->length, sizeof(list->c[0]), chain_sort);
while (curr < list->length - 1) {
- check_t *current = &list->c[curr];
- size_t count = curr;
- char *cur_name = bam_get_qname(current->b);
- int current_paired = (current->b->core.flag & BAM_FPAIRED) && !(current->b->core.flag & BAM_FMUNMAP);
+ check_t *base = &list->c[curr];
+ char *base_name = bam_get_qname(base->b);
+ int end_name_match = curr;
- while (++count < list->length && (list->c[count].x - current->x <= param->opt_dist)) {
- // while close enough along the x coordinate
- check_t *chk = &list->c[count];
+ // find the end of the matching name parts
+ while (++end_name_match < list->length) {
+ check_t *chk = &list->c[end_name_match];
- if (current->opt && chk->opt)
- continue;
+ if ((base->len == chk->len) && memcmp(base_name + base->beg, bam_get_qname(chk->b) + chk->beg, base->len) != 0)
+ break;
+ }
- // if both are already optical duplicates there is no need to check again, otherwise...
+ while (curr < end_name_match) {
+ size_t count = curr;
+ check_t *current = &list->c[curr];
+ int current_paired = (current->b->core.flag & BAM_FPAIRED) && !(current->b->core.flag & BAM_FMUNMAP);
- long ydiff;
+ while (++count < end_name_match && (list->c[count].x - current->x <= param->opt_dist)) {
+ // while close enough along the x coordinate
+ check_t *chk = &list->c[count];
- if (current->y > chk->y) {
- ydiff = current->y - chk->y;
- } else {
- ydiff = chk->y - current->y;
- }
+ if (current->opt && chk->opt)
+ continue;
- if (ydiff > param->opt_dist)
- continue;
+ long ydiff;
- // the number are right, check the names
- if (strncmp(cur_name + current->beg, bam_get_qname(chk->b) + chk->beg, current->end - current->beg) != 0)
- continue;
+ if (current->y > chk->y) {
+ ydiff = current->y - chk->y;
+ } else {
+ ydiff = chk->y - current->y;
+ }
- // optical duplicates
- int chk_dup = 0;
- int chk_paired = (chk->b->core.flag & BAM_FPAIRED) && !(chk->b->core.flag & BAM_FMUNMAP);
+ if (ydiff > param->opt_dist)
+ continue;
- if (current_paired != chk_paired) {
- if (!chk_paired) {
- // chk is single vs pair, this is a dup.
- chk_dup = 1;
- }
- } else {
- // do it by scores
- int64_t cur_score, chk_score;
+ // optical duplicates
+ int chk_dup = 0;
+ int chk_paired = (chk->b->core.flag & BAM_FPAIRED) && !(chk->b->core.flag & BAM_FMUNMAP);
- if ((current->b->core.flag & BAM_FQCFAIL) != (chk->b->core.flag & BAM_FQCFAIL)) {
- if (current->b->core.flag & BAM_FQCFAIL) {
- cur_score = 0;
- chk_score = 1;
- } else {
- cur_score = 1;
- chk_score = 0;
+ if (current_paired != chk_paired) {
+ if (!chk_paired) {
+ // chk is single vs pair, this is a dup.
+ chk_dup = 1;
}
} else {
- cur_score = current->score;
- chk_score = chk->score;
+ // do it by scores
+ int64_t cur_score, chk_score;
- if (current_paired) {
- // they are pairs so add mate scores.
- chk_score += chk->mate_score;
- cur_score += current->mate_score;
+ if ((current->b->core.flag & BAM_FQCFAIL) != (chk->b->core.flag & BAM_FQCFAIL)) {
+ if (current->b->core.flag & BAM_FQCFAIL) {
+ cur_score = 0;
+ chk_score = 1;
+ } else {
+ cur_score = 1;
+ chk_score = 0;
+ }
+ } else {
+ cur_score = current->score;
+ chk_score = chk->score;
+
+ if (current_paired) {
+ // they are pairs so add mate scores.
+ chk_score += chk->mate_score;
+ cur_score += current->mate_score;
+ }
}
- }
- if (cur_score == chk_score) {
- if (strcmp(bam_get_qname(chk->b), cur_name) < 0) {
- chk_score++;
- } else {
- chk_score--;
+ if (cur_score == chk_score) {
+ if (strcmp(bam_get_qname(chk->b), bam_get_qname(current->b)) < 0) {
+ chk_score++;
+ } else {
+ chk_score--;
+ }
}
- }
- if (cur_score > chk_score) {
- chk_dup = 1;
+ if (cur_score > chk_score) {
+ chk_dup = 1;
+ }
}
- }
- if (chk_dup) {
- // the duplicate is the optical duplicate
- if (!chk->opt) { // only change if not already an optical duplicate
- if (optical_retag(param, dup_hash, chk->b, chk_paired, stats)) {
- ret = -1;
- goto fail;
- }
+ if (chk_dup) {
+ // the duplicate is the optical duplicate
+ if (!chk->opt) { // only change if not already an optical duplicate
+ if (optical_retag(param, dup_hash, chk->b, chk_paired, stats)) {
+ ret = -1;
+ goto fail;
+ }
- chk->opt = 1;
- }
- } else {
- if (!current->opt) {
- if (optical_retag(param, dup_hash, current->b, current_paired, stats)) {
- ret = -1;
- goto fail;
+ chk->opt = 1;
}
+ } else {
+ if (!current->opt) {
+ if (optical_retag(param, dup_hash, current->b, current_paired, stats)) {
+ ret = -1;
+ goto fail;
+ }
- current->opt = 1;
+ current->opt = 1;
+ }
}
}
- }
- curr++;
+ curr++;
+ }
}
fail:
int opt;
int beg;
int end;
+ int len;
} check_t;
typedef struct {
return ret;
}
- if (strncmp(original + o_beg, duplicate + d_beg, o_end - o_beg) == 0) {
+ int o_len = o_end - o_beg;
+ int d_len = d_end - d_beg;
+
+ if ((o_len == d_len) && memcmp(original + o_beg, duplicate + d_beg, o_len) == 0) {
long xdiff, ydiff;
if (ox > dx) {
return ret;
}
- if (strncmp(name + o_beg, duplicate + d_beg, o_end - o_beg) == 0) {
+ int o_len = o_end - o_beg;
+ int d_len = d_end - d_beg;
+
+ if ((o_len == d_len) && memcmp(name + o_beg, duplicate + d_beg, o_len) == 0) {
// the initial parts match, look at the numbers
long xdiff, ydiff;
c->y = dy;
c->beg = d_beg;
c->end = d_end;
+ c->len = d_end - d_beg;
return ret;
}
}
-static int xcoord_sort(const void *a, const void *b) {
+static int chain_sort(const void *a, const void *b) {
check_t *ac = (check_t *) a;
check_t *bc = (check_t *) b;
+ int ret;
+
+ if ((ret = ac->len - bc->len))
+ return ret;
+ else if ((ret = memcmp(bam_get_qname(ac->b) + ac->beg, bam_get_qname(bc->b) + bc->beg, ac->len)))
+ return ret;
return (ac->x - bc->x);
}
int ret = 0;
size_t curr = 0;
- qsort(list->c, list->length, sizeof(list->c[0]), xcoord_sort);
+ qsort(list->c, list->length, sizeof(list->c[0]), chain_sort);
while (curr < list->length - 1) {
- check_t *current = &list->c[curr];
- size_t count = curr;
- char *cur_name = bam_get_qname(current->b);
- int current_paired = (current->b->core.flag & BAM_FPAIRED) && !(current->b->core.flag & BAM_FMUNMAP);
+ check_t *base = &list->c[curr];
+ char *base_name = bam_get_qname(base->b);
+ int end_name_match = curr;
- while (++count < list->length && (list->c[count].x - current->x <= param->opt_dist)) {
- // while close enough along the x coordinate
- check_t *chk = &list->c[count];
+ // find the end of the matching name parts
+ while (++end_name_match < list->length) {
+ check_t *chk = &list->c[end_name_match];
- if (current->opt && chk->opt)
- continue;
+ if ((base->len == chk->len) && memcmp(base_name + base->beg, bam_get_qname(chk->b) + chk->beg, base->len) != 0)
+ break;
+ }
- // if both are already optical duplicates there is no need to check again, otherwise...
+ while (curr < end_name_match) {
+ size_t count = curr;
+ check_t *current = &list->c[curr];
+ int current_paired = (current->b->core.flag & BAM_FPAIRED) && !(current->b->core.flag & BAM_FMUNMAP);
- long ydiff;
+ while (++count < end_name_match && (list->c[count].x - current->x <= param->opt_dist)) {
+ // while close enough along the x coordinate
+ check_t *chk = &list->c[count];
- if (current->y > chk->y) {
- ydiff = current->y - chk->y;
- } else {
- ydiff = chk->y - current->y;
- }
+ if (current->opt && chk->opt)
+ continue;
- if (ydiff > param->opt_dist)
- continue;
+ long ydiff;
- // the number are right, check the names
- if (strncmp(cur_name + current->beg, bam_get_qname(chk->b) + chk->beg, current->end - current->beg) != 0)
- continue;
+ if (current->y > chk->y) {
+ ydiff = current->y - chk->y;
+ } else {
+ ydiff = chk->y - current->y;
+ }
- // optical duplicates
- int chk_dup = 0;
- int chk_paired = (chk->b->core.flag & BAM_FPAIRED) && !(chk->b->core.flag & BAM_FMUNMAP);
+ if (ydiff > param->opt_dist)
+ continue;
- if (current_paired != chk_paired) {
- if (!chk_paired) {
- // chk is single vs pair, this is a dup.
- chk_dup = 1;
- }
- } else {
- // do it by scores
- int64_t cur_score, chk_score;
+ // optical duplicates
+ int chk_dup = 0;
+ int chk_paired = (chk->b->core.flag & BAM_FPAIRED) && !(chk->b->core.flag & BAM_FMUNMAP);
- if ((current->b->core.flag & BAM_FQCFAIL) != (chk->b->core.flag & BAM_FQCFAIL)) {
- if (current->b->core.flag & BAM_FQCFAIL) {
- cur_score = 0;
- chk_score = 1;
- } else {
- cur_score = 1;
- chk_score = 0;
+ if (current_paired != chk_paired) {
+ if (!chk_paired) {
+ // chk is single vs pair, this is a dup.
+ chk_dup = 1;
}
} else {
- cur_score = current->score;
- chk_score = chk->score;
+ // do it by scores
+ int64_t cur_score, chk_score;
- if (current_paired) {
- // they are pairs so add mate scores.
- chk_score += chk->mate_score;
- cur_score += current->mate_score;
+ if ((current->b->core.flag & BAM_FQCFAIL) != (chk->b->core.flag & BAM_FQCFAIL)) {
+ if (current->b->core.flag & BAM_FQCFAIL) {
+ cur_score = 0;
+ chk_score = 1;
+ } else {
+ cur_score = 1;
+ chk_score = 0;
+ }
+ } else {
+ cur_score = current->score;
+ chk_score = chk->score;
+
+ if (current_paired) {
+ // they are pairs so add mate scores.
+ chk_score += chk->mate_score;
+ cur_score += current->mate_score;
+ }
}
- }
- if (cur_score == chk_score) {
- if (strcmp(bam_get_qname(chk->b), cur_name) < 0) {
- chk_score++;
- } else {
- chk_score--;
+ if (cur_score == chk_score) {
+ if (strcmp(bam_get_qname(chk->b), bam_get_qname(current->b)) < 0) {
+ chk_score++;
+ } else {
+ chk_score--;
+ }
}
- }
- if (cur_score > chk_score) {
- chk_dup = 1;
+ if (cur_score > chk_score) {
+ chk_dup = 1;
+ }
}
- }
- if (chk_dup) {
- // the duplicate is the optical duplicate
- if (!chk->opt) { // only change if not already an optical duplicate
- if (optical_retag(param, dup_hash, chk->b, chk_paired, stats)) {
- ret = -1;
- goto fail;
- }
+ if (chk_dup) {
+ // the duplicate is the optical duplicate
+ if (!chk->opt) { // only change if not already an optical duplicate
+ if (optical_retag(param, dup_hash, chk->b, chk_paired, stats)) {
+ ret = -1;
+ goto fail;
+ }
- chk->opt = 1;
- }
- } else {
- if (!current->opt) {
- if (optical_retag(param, dup_hash, current->b, current_paired, stats)) {
- ret = -1;
- goto fail;
+ chk->opt = 1;
}
+ } else {
+ if (!current->opt) {
+ if (optical_retag(param, dup_hash, current->b, current_paired, stats)) {
+ ret = -1;
+ goto fail;
+ }
- current->opt = 1;
+ current->opt = 1;
+ }
}
}
- }
- curr++;
+ curr++;
+ }
}
fail:
/* bam_mate.c -- fix mate pairing information and clean up flags.
- Copyright (C) 2009, 2011-2017, 2019, 2022 Genome Research Ltd.
+ Copyright (C) 2009, 2011-2017, 2019, 2022, 2024 Genome Research Ltd.
Portions copyright (C) 2011 Broad Institute.
Portions copyright (C) 2012 Peter Cock, The James Hutton Institute.
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
+#include <ctype.h>
#include "htslib/thread_pool.h"
#include "sam_opts.h"
#include "htslib/kstring.h"
* single Reads:
* -if pos == 0 (1 based), tid == -1, or UNMAPPED then set UNMAPPED, pos = 0,
* tid = -1
- * -clear bad flags (PAIRED, MREVERSE, PROPER_PAIR)
+ * -clear bad flags (MREVERSE, PROPER_PAIR)
* -set mpos = 0 (1 based), mtid = -1 and isize = 0
* -write to output
* Paired Reads:
return 0;
}
+// Look for 3 tags in one pass, for efficiencies sake.
+// We also convert the draft tags Mm and Ml to MM and ML here.
+static inline void find_tags(bam1_t *b,
+ char *t1, uint8_t **t1p,
+ char *t2, uint8_t **t2p,
+ char *t3, uint8_t **t3p) {
+ *t1p = *t2p = *t3p = NULL;
+ uint8_t *aux = bam_aux_first(b);
+
+ while (aux) {
+ if (aux[-2] == t1[0] && toupper(aux[-1]) == t1[1]) {
+ *t1p = aux;
+ if (islower(aux[-1]))
+ aux[-1] = t1[1];
+ } else if (aux[-2] == t2[0] && toupper(aux[-1]) == t2[1]) {
+ *t2p = aux;
+ if (islower(aux[-1]))
+ aux[-1] = t2[1];
+ } else if (aux[-2] == t3[0] && toupper(aux[-1]) == t3[1]) {
+ *t3p = aux;
+ if (islower(aux[-1]))
+ aux[-1] = t3[1];
+ }
+ aux = bam_aux_next(b, aux);
+ }
+}
+
+// Return 5' and 3' CIGAR hard-clip counts
+static inline void hard_clips(bam1_t *b, int *end5, int *end3) {
+ uint32_t *cigar = bam_get_cigar(b);
+ int ncigar = b->core.n_cigar;
+ int endL = 0, endR = 0, nh = 0;
+
+ if (ncigar && bam_cigar_op(cigar[0]) == BAM_CHARD_CLIP)
+ endL = bam_cigar_oplen(cigar[0]), nh=1;
+ if (ncigar > nh && bam_cigar_op(cigar[ncigar-1]) == BAM_CHARD_CLIP)
+ endR = bam_cigar_oplen(cigar[ncigar-1]);
+
+ if (b->core.flag & BAM_FREVERSE) {
+ *end5 = endR;
+ *end3 = endL;
+ } else {
+ *end5 = endL;
+ *end3 = endR;
+ }
+}
+
+// Get MM, ML and MN tags, and 5' and 3' hard-clip lengths.
+// MNi is integer copy of MN, or -1 if absent/invalid
+void get_mod_info(bam1_t *b, uint8_t **MM, uint8_t **ML, uint8_t **MN,
+ int *MNi, int *end5, int *end3) {
+ find_tags(b, "MM", MM, "ML", ML, "MN", MN);
+ if (*MN) {
+ int save_errno = errno;
+ errno = 0;
+ *MNi = bam_aux2i(*MN);
+ if (errno == EINVAL)
+ *MNi = -1;
+ errno = save_errno;
+ } else {
+ *MNi = -1;
+ }
+
+ if (*MM)
+ hard_clips(b, end5, end3);
+ else
+ *end5 = *end3 = 0; // don't need if MM not found
+}
+
+typedef struct MM_state {
+ // tags found on "pre" BAM
+ uint8_t *MM, *ML, *MN;
+} MM_state;
+
+uint8_t *MN_enc(uint8_t *tag, uint32_t n) {
+ if (n > UINT16_MAX) {
+ tag[0] = 'I';
+ i32_to_le(n, tag+1);
+ tag += 5;
+ } else if (n > UINT8_MAX) {
+ tag[0] = 'S';
+ i16_to_le(n, tag+1);
+ tag += 3;
+ } else {
+ *tag++ = 'C';
+ *tag++ = n;
+ }
+
+ return tag;
+}
+
+// Trim 5'/3' bases off MM and ML tags, using a previous sequence as a guide.
+int trim_MM(bam1_t *pre, bam1_t *cur, int end5, int end3,
+ uint8_t *MM, uint8_t *ML, uint8_t *MN) {
+ // Count number of bases
+ int counts5[16] = {0}, counts3[16] = {0};
+
+ uint8_t *seq = bam_get_seq(pre);
+ int i;
+ for (i = 0; i < end5; i++)
+ counts5[bam_seqi(seq, i)]++;
+ memcpy(counts3, counts5, 16 * sizeof(*counts3));
+ for (; i < pre->core.l_qseq - end3; i++)
+ counts3[bam_seqi(seq, i)]++;
+
+ // "p" is position in pre.
+ // "q" is position in cur.
+ // Hence move up "p" to start and copy from there to "q".
+ uint8_t *MMp, *MLp, *MMq = NULL, *MLq = NULL;
+ if (ML && ML[0] == 'B' && ML[1] == 'C') {
+ MLp = ML+6;
+ } else {
+ ML = MLp = NULL;
+ }
+ MMq = MM+1;
+ MLq = MLp;
+ for (MMp = MM+1; *MMp; ) {
+ int fundamental = seq_nt16_table[*MMp];
+ while (*MMp && *MMp != ',')
+ *MMq++ = *MMp++;
+ if (*MMp)
+ *MMq++ = *MMp++;
+
+ // Now on comma separated list for MM and BC array for ML. Skip
+ int n = 0;
+ while (*MMp != ';' && n < counts5[fundamental]) {
+ char *endptr;
+ long delta = strtol((char *)MMp, &endptr, 10);
+ if (counts5[fundamental] - n > delta) {
+ // Skip entire delta in MM and ML.
+ // Eg counts[]=10, MM=3,10 ML=<10><20> => MM=10 ML=<20>
+ n += delta+1;
+ if(ML) MLp++;
+ } else if (counts3[fundamental] > counts5[fundamental]) {
+ // Shrink delta, writing MM and ML is unchanged.
+ // Eg counts[]=3, MM=10,4 ML=<10><20> => MM=7,4 ML=<10><20>
+ char num[50];
+ int l = sprintf(num, "%ld",
+ delta - (counts5[fundamental]-n));
+ memcpy((char *)MMq, num, l);
+ MMq += l;
+ *MMq++ = *endptr;
+ n += delta+1;
+ if (ML)
+ *MLq++ = *MLp++;
+ } else {
+ // next base mod is on boundary of 3' clip point
+ break;
+ }
+
+ MMp = (uint8_t *)endptr;
+ if (*MMp != ',')
+ // error? if not ; also?
+ break;
+ MMp++;
+ }
+
+ // Copy
+ while (*MMp != ';' && n < counts3[fundamental]) {
+ char *endptr;
+ long delta = strtol((char *)MMp, &endptr, 10);
+ if (counts3[fundamental] - n > delta) {
+ // Copy entire delta in MM and ML including [,;]
+ memmove(MMq, MMp, (uint8_t *)endptr - MMp + 1);
+ MMq += (uint8_t *)endptr - MMp + 1;
+ n += delta+1;
+ if (ML)
+ *MLq++ = *MLp++;
+ } else {
+ // Next mod is into 3' cutoff, so can terminate MM/ML now
+ n = counts3[fundamental];
+ if (ML)
+ MLp++;
+ }
+
+ MMp = (uint8_t *)endptr;
+ if (*MMp != ',')
+ break;
+ MMp++;
+ }
+
+ // Skip
+ while (*MMp && *MMp != ';') {
+ while (*MMp && *MMp != ',' && *MMp != ';')
+ MMp++;
+ if (*MMp == ',')
+ MMp++;
+
+ if (ML)
+ MLp++;
+ }
+ MMq[-1] = ';'; // replaces , with ; if clipping right
+ if (*MMp)
+ MMp++;
+ }
+
+ MMp++; // skip nul
+ *MMq++ = 0;
+
+ // Adjust ML B array length
+ if (ML)
+ u32_to_le(MLq-(ML+6), ML+2);
+
+ // Move MM and ML down to include their MM:Z and ML:B bits
+ if (MM) MM-=2;
+ if (ML) ML-=2;
+
+ // Now MM/ML are start of tags, MMq/MLq are ends of edited tags,
+ // and MMp/MLp are ends of original tags. Walk through tags taking up
+ // any gaps
+ //
+ // Eg XXXXXXmmmmm--YYYlllll-ZZ (m and l are edited MM and ML tags)
+ // => XXXXXXmmmmmYYYlllllZZ
+
+ uint8_t *tag = bam_get_aux(cur), *tag_end = cur->data + cur->l_data;
+ uint8_t *to = tag;
+ while (tag && tag < tag_end) {
+ if (tag[0] == 'M' && (tag[1] == 'M' || tag[1] == 'm')) {
+ // Slow but easy
+ memmove(to, MM, MMq-MM); // length of new tag
+ to += MMq-MM;
+ tag = MMp; // size of old tag
+ } else if (tag[0] == 'M' && (tag[1] == 'L' || tag[1] == 'l')) {
+ memmove(to, ML, MLq-ML);
+ to += MLq-ML;
+ tag = MLp;
+ } else if (tag[0] == 'M' && tag[1] == 'N') {
+ tag = bam_aux_next(cur, tag+2);
+ // Skip it as we'll overwrite this later, although this
+ // does change the tag order. Instead we could do:
+ //
+ // *to++ = 'M';
+ // *to++ = 'N';
+ // to = MN_enc(to, cur->core.l_qseq);
+ } else {
+ // Want aux_skip, but it's private.
+ // So we use bam_aux_next with work-arounds. :(
+ uint8_t *from = tag;
+ tag = bam_aux_next(cur, tag+2);
+ tag = tag ? tag-2 : tag_end;
+ memmove(to, from, tag-from);
+ to += tag-from;
+ }
+ }
+ cur->l_data = to - cur->data;
+
+ return 0;
+}
+
+// Removes base modification tags: MM, ML and MN.
+// This is more efficient than a series of bam_aux_remove and
+// bam_aux_find calls, as the previous removes shuffle the tags we've
+// previously found. However it's still not optimal.
+void delete_mod_tags(bam1_t *b) {
+ uint8_t *tag = bam_aux_first(b), *next;
+ uint8_t *to = tag;
+ while (tag) {
+ next = bam_aux_next(b, tag);
+ if (tag[-2] == 'M' &&
+ (tag[-1] == 'M' || tag[-1] == 'm' ||
+ tag[-1] == 'L' || tag[-1] == 'l' ||
+ tag[-1] == 'N')) {
+ // Skip. Equivalent to bam_aux_remove without multiple passes
+ } else {
+ // Copy. All these +/-2s are an annoyance caused by the
+ // tag iterator pointing to the byte after the 2-letter code
+ uint8_t *end = next ? next : b->data + b->l_data + 2;
+ if (tag != to)
+ memmove(to-2, tag-2, end-tag);
+ to += end-tag;
+ }
+ tag = next;
+ }
+
+ b->l_data = (to-2) - b->data;
+}
+
+int validate_MM(bam1_t *b, hts_base_mod_state *state) {
+ hts_base_mod mods[10];
+ int n, pos;
+ while ((n = bam_next_basemod(b, state, mods, 10, &pos)) > 0) {
+ // bam_next_basemod will trigger MM out-of-bound checks
+ }
+ return n;
+}
+
+// Fix base modification tags MM, ML and MN.
+// For supplementary-style alignments we may have hard-clipped the sequence
+// and just duplicated the MM/ML tags. Use the primary alignment to get the
+// clipped sequence so we can trim MM/ML accordingly.
+//
+// We call this first on primary reads with pre == NULL. This caches
+// MM and ML data into MM_state.
+//
+// We then call it again on secondary and/or supplementary data with
+// pre == the primary record and pass in the associated state. This then
+// validates MM/ML/MN match, and if not adjusts them if they have hard-clips
+// which yields consistent data.
+//
+// TODO: add sanity check on counts of base types and MM tag to ensure it's
+// possible. We can do this post-trimming, so we sanitize everything.
+//
+// Returns 0 on success,
+// -1 on failure
+int fix_MM(bam1_t *pre, bam1_t *cur, MM_state *state) {
+ int end5, end3;
+ int MNi = 0; // MN of -1 is used as indicator for no valid mods
+
+ if (!pre && state) {
+ // First time we've see this name.
+ // Look for base modification tags and sanity check.
+ get_mod_info(cur, &state->MM, &state->ML, &state->MN, &MNi,
+ &end5, &end3);
+ if (!state->MM) {
+ delete_mod_tags(cur);
+ return 0;
+ }
+
+ if (!end5 && !end3 && MNi <= 0) {
+ // No MN tag, but also no clipping. Assume MM is valid
+ if (cur->core.l_qseq)
+ if (bam_aux_update_int(cur, "MN", cur->core.l_qseq) < 0)
+ return -1;
+ } else if ((end3 || end5) && cur->core.l_qseq != MNi) {
+ // We have hard clips and MN tag, but the MN tag doesn't match
+ // observed sequence length so it appears the hard-clipping
+ // happened after base-mods called without updating.
+ // Fail as this is a primary read.
+ delete_mod_tags(cur);
+ }
+ // Otherwise we assume the base modifications are correct
+
+ } else if (state) {
+ // A supplementary or secondary alignment with known primary
+ uint8_t *cur_MM = NULL, *cur_ML = NULL, *cur_MN = NULL;
+ MNi = -1;
+ get_mod_info(cur, &cur_MM, &cur_ML, &cur_MN, &MNi, &end5, &end3);
+
+ if (!cur_MM) {
+ delete_mod_tags(cur);
+ return 0;
+ }
+
+ // Does MN match seq length? If so, we believe it's already valid
+ if (MNi == cur->core.l_qseq)
+ goto validate;
+
+ // Length mismatch and/or no known length, so check vs full seq.
+ if (pre->core.l_qseq != cur->core.l_qseq + end3 + end5) {
+ delete_mod_tags(cur);
+ return 0;
+ } else if (end5 || end3) {
+ if (MNi < 0 || MNi == pre->core.l_qseq)
+ trim_MM(pre, cur, end5, end3, cur_MM, cur_ML, cur_MN);
+ } // else no hard clips so MM is already valid
+
+ // Set MN so we've validated it, provided seq isn't "*".
+ // inefficient, but minimal compared to everything else
+ if (cur->core.l_qseq)
+ if (bam_aux_update_int(cur, "MN", cur->core.l_qseq) < 0)
+ return -1;
+ }
+
+ validate:
+ ;
+
+ // Also validate MM length matches sequence length. This mirrors the
+ // logic in htslib/sam_mods.c.
+ // For now we take the inefficient approach of using bam_parse_basemod2.
+ // Inefficient, but robust.
+ hts_base_mod_state *mst = hts_base_mod_state_alloc();
+ if (!mst)
+ return -1;
+
+ enum htsLogLevel lvl = hts_get_log_level();
+ hts_set_log_level(HTS_LOG_OFF);
+ if (bam_parse_basemod(cur, mst) < 0)
+ // Maybe we want hts_log_warning still though?
+ delete_mod_tags(cur);
+ if (validate_MM(cur, mst) < 0)
+ delete_mod_tags(cur);
+ hts_set_log_level(lvl);
+ hts_base_mod_state_free(mst);
+
+ return 0;
+}
+
+// Ensure the b[] array is at least n.
+// Returns 0 on success,
+// -1 on failure
+static int grow_b_array(bam1_t **b, int *ba, int n) {
+ if (n < *ba)
+ return 0;
+
+ bam1_t *bnew = realloc(*b, (n+=10) * sizeof(**b));
+ if (!bnew)
+ return -1;
+ *b = bnew;
+
+ // bam_init1 equivalent
+ int i;
+ for (i = *ba; i < n; i++)
+ memset(&(*b)[i], 0, sizeof(bam1_t));
+
+ *b = bnew;
+ *ba = n;
+
+ return 0;
+}
+
+// We have b[0]..b[bn-1] entries all from the same template (qname)
+typedef struct {
+ bam1_t *b;
+ int n, ba; // number used and number allocated
+ int b_next; // b[b_next] for start of next set, -1 if unset
+ int eof; // marker for having seen eof
+} bam_set;
+
+// Fetches a new batch of BAM records all containing the same name.
+// NB: we cache the last (non-matching) name in b[n], so we can use it to
+// start the next batch.
+// Returns the number of records on success,
+// <0 on failure or EOF (sam_read1 return vals)
+static int next_template(samFile *in, sam_hdr_t *header, bam_set *bs,
+ int sanitize_flags) {
+ int result;
+
+ if (bs->eof)
+ return -1;
+
+ // First time through, prime the template name
+ if (bs->b_next < 0) {
+ if (grow_b_array(&bs->b, &bs->ba, 1) < 0)
+ return -2;
+ result = sam_read1(in, header, &bs->b[0]);
+ if (result < 0)
+ return result;
+ if (bam_sanitize(header, &bs->b[0], sanitize_flags) < 0)
+ return -2;
+ } else {
+ // Otherwise use the previous template name read
+ bam1_t btmp = bs->b[0];
+ bs->b[0] = bs->b[bs->b_next];
+ bs->b[bs->b_next] = btmp; // For ->{,l_,m_}data
+ }
+ bs->n = 1;
+
+ // Now keep reading until we find a read that mismatches or we hit eof.
+ char *name = bam_get_qname(&bs->b[0]);
+ for (;;) {
+ if (grow_b_array(&bs->b, &bs->ba, bs->n+1) < 0)
+ return -2;
+
+ result = sam_read1(in, header, &bs->b[bs->n]);
+ if (result < -1)
+ return result;
+
+ if (result < 0) {
+ bs->eof = 1;
+ bs->b_next = -1;
+ break;
+ } else {
+ if (bam_sanitize(header, &bs->b[bs->n], sanitize_flags) < 0)
+ return -2;
+
+ bs->b_next = bs->n;
+ if (strcmp(name, bam_get_qname(&bs->b[bs->n])) != 0)
+ break;
+ }
+
+ bs->n++;
+ }
+
+ return bs->n;
+}
+
// currently, this function ONLY works if each read has one hit
+//
+// Returns 0 on success,
+// >0 on failure
static int bam_mating_core(samFile *in, samFile *out, int remove_reads,
int proper_pair_check, int add_ct,
int do_mate_scoring, char *arg_list, int no_pg,
- int sanitize_flags)
+ int sanitize_flags, int base_mods)
{
sam_hdr_t *header;
- bam1_t *b[2] = { NULL, NULL };
- int curr, has_prev, result;
- hts_pos_t pre_end = 0, cur_end = 0;
+ int result, n;
kstring_t str = KS_INITIALIZE;
+ bam_set bs = {NULL, 0, 0, -1, 0};
header = sam_hdr_read(in);
if (header == NULL) {
if (sam_hdr_write(out, header) < 0) goto write_fail;
- b[0] = bam_init1();
- b[1] = bam_init1();
- curr = 0; has_prev = 0;
- while ((result = sam_read1(in, header, b[curr])) >= 0) {
- bam1_t *cur = b[curr], *pre = b[1-curr];
- if (bam_sanitize(header, cur, sanitize_flags) < 0)
- goto fail;
- if (cur->core.flag & BAM_FSECONDARY)
- {
- if ( !remove_reads ) {
- if (sam_write1(out, header, cur) < 0) goto write_fail;
+ // Iterate template by template fetching bs->n records at a time
+ while ((result = next_template(in, header, &bs, sanitize_flags)) >= 0) {
+ bam1_t *cur = NULL, *pre = NULL, *rnum[2] = {NULL, NULL};
+ int prev = -1, curr = -1;
+ hts_pos_t pre_end = 0, cur_end = 0;
+
+ // Find and fix up primary alignments
+ MM_state state[2];
+ for (n = 0; n < bs.n; n++) {
+ int is_r2 = (bs.b[n].core.flag & BAM_FREAD2) != 0;
+ if (bs.b[n].core.flag & (BAM_FSECONDARY | BAM_FSUPPLEMENTARY))
+ continue;
+
+ if (base_mods)
+ if (fix_MM(NULL, &bs.b[n], &state[is_r2]) < 0)
+ goto fail;
+
+ if (!pre) {
+ pre = &bs.b[prev = n];
+ rnum[(pre->core.flag & BAM_FREAD2) != 0] = pre;
+
+ pre_end = (pre->core.flag & BAM_FUNMAP) == 0
+ ? bam_endpos(pre) : 0;
+ continue;
}
- continue; // skip secondary alignments
- }
- if (cur->core.flag & BAM_FSUPPLEMENTARY)
- {
- if (sam_write1(out, header, cur) < 0) goto write_fail;
- continue; // pass supplementary alignments through unchanged (TODO:make them match read they came from)
- }
- if ((cur->core.flag&BAM_FUNMAP) == 0) // If mapped calculate end
- {
- cur_end = bam_endpos(cur);
- }
- if (has_prev) { // do we have a pair of reads to examine?
- if (strcmp(bam_get_qname(cur), bam_get_qname(pre)) == 0) { // identical pair name
- pre->core.flag |= BAM_FPAIRED;
- cur->core.flag |= BAM_FPAIRED;
- if (sync_mate(pre, cur)) goto fail;
-
- if (pre->core.tid == cur->core.tid && !(cur->core.flag&(BAM_FUNMAP|BAM_FMUNMAP))
- && !(pre->core.flag&(BAM_FUNMAP|BAM_FMUNMAP))) // if safe set TLEN/ISIZE
- {
- hts_pos_t cur5, pre5;
- cur5 = (cur->core.flag&BAM_FREVERSE)? cur_end : cur->core.pos;
- pre5 = (pre->core.flag&BAM_FREVERSE)? pre_end : pre->core.pos;
- cur->core.isize = pre5 - cur5; pre->core.isize = cur5 - pre5;
- } else cur->core.isize = pre->core.isize = 0;
- if (add_ct) bam_template_cigar(pre, cur, &str);
- // TODO: Add code to properly check if read is in a proper pair based on ISIZE distribution
- if (proper_pair_check && !plausibly_properly_paired(pre,cur)) {
- pre->core.flag &= ~BAM_FPROPER_PAIR;
- cur->core.flag &= ~BAM_FPROPER_PAIR;
- }
+ // Note, more than 2 primary alignments will use 'curr' as last
+ cur = &bs.b[curr = n];
+ rnum[(cur->core.flag & BAM_FREAD2) != 0] = cur;
+ cur_end = (cur->core.flag & BAM_FUNMAP) == 0
+ ? bam_endpos(cur) : 0;
+
+ pre->core.flag |= BAM_FPAIRED;
+ cur->core.flag |= BAM_FPAIRED;
+ if (sync_mate(pre, cur))
+ goto fail;
+
+ // If safe set TLEN/ISIZE
+ if (pre->core.tid == cur->core.tid
+ && !(cur->core.flag & (BAM_FUNMAP | BAM_FMUNMAP))
+ && !(pre->core.flag & (BAM_FUNMAP | BAM_FMUNMAP))) {
+ hts_pos_t cur5, pre5;
+ cur5 = (cur->core.flag & BAM_FREVERSE)
+ ? cur_end
+ : cur->core.pos;
+ pre5 = (pre->core.flag & BAM_FREVERSE)
+ ? pre_end
+ : pre->core.pos;
+ cur->core.isize = pre5 - cur5;
+ pre->core.isize = cur5 - pre5;
+ } else {
+ cur->core.isize = pre->core.isize = 0;
+ }
- if (do_mate_scoring) {
- if ((add_mate_score(pre, cur) == -1) || (add_mate_score(cur, pre) == -1)) {
- fprintf(stderr, "[bam_mating_core] ERROR: unable to add mate score.\n");
- goto fail;
- }
- }
+ if (add_ct)
+ bam_template_cigar(pre, cur, &str);
- // Write out result
- if ( !remove_reads ) {
- if (sam_write1(out, header, pre) < 0) goto write_fail;
- if (sam_write1(out, header, cur) < 0) goto write_fail;
- } else {
- // If we have to remove reads make sure we do it in a way that doesn't create orphans with bad flags
- if(pre->core.flag&BAM_FUNMAP) cur->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if(cur->core.flag&BAM_FUNMAP) pre->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if(!(pre->core.flag&BAM_FUNMAP)) {
- if (sam_write1(out, header, pre) < 0) goto write_fail;
- }
- if(!(cur->core.flag&BAM_FUNMAP)) {
- if (sam_write1(out, header, cur) < 0) goto write_fail;
- }
- }
- has_prev = 0;
- } else { // unpaired? clear bad info and write it out
- pre->core.mtid = -1; pre->core.mpos = -1; pre->core.isize = 0;
- pre->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if ( !remove_reads || !(pre->core.flag&BAM_FUNMAP) ) {
- if (sam_write1(out, header, pre) < 0) goto write_fail;
+ // TODO: Add code to properly check if read is in a proper
+ // pair based on ISIZE distribution
+ if (proper_pair_check && !plausibly_properly_paired(pre,cur)) {
+ pre->core.flag &= ~BAM_FPROPER_PAIR;
+ cur->core.flag &= ~BAM_FPROPER_PAIR;
+ }
+
+ if (do_mate_scoring) {
+ if ((add_mate_score(pre, cur) == -1) ||
+ (add_mate_score(cur, pre) == -1)) {
+ fprintf(stderr, "[bam_mating_core] ERROR: "
+ "unable to add mate score.\n");
+ goto fail;
}
}
- } else has_prev = 1;
- curr = 1 - curr;
- pre_end = cur_end;
- }
- if (result < -1) goto read_fail;
- if (has_prev && !remove_reads) { // If we still have a BAM in the buffer it must be unpaired
- bam1_t *pre = b[1-curr];
- if (pre->core.tid < 0 || pre->core.pos < 0 || pre->core.flag&BAM_FUNMAP) { // If unmapped
- pre->core.flag |= BAM_FUNMAP;
- pre->core.tid = -1;
- pre->core.pos = -1;
+
+ // If we have to remove reads make sure we do it in a way that
+ // doesn't create orphans with bad flags
+ if (remove_reads) {
+ if (pre->core.flag&BAM_FUNMAP)
+ cur->core.flag &=
+ ~(BAM_FMREVERSE|BAM_FPROPER_PAIR);
+ if (cur->core.flag&BAM_FUNMAP)
+ pre->core.flag &=
+ ~(BAM_FMREVERSE|BAM_FPROPER_PAIR);
+ }
}
- pre->core.mtid = -1; pre->core.mpos = -1; pre->core.isize = 0;
- pre->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if (sam_write1(out, header, pre) < 0) goto write_fail;
+ // Handle unpaired primary data
+ if (!cur && pre) {
+ pre->core.mtid = -1;
+ pre->core.mpos = -1;
+ pre->core.isize = 0;
+ pre->core.flag &= ~(BAM_FMREVERSE|BAM_FPROPER_PAIR);
+ }
+
+ // Now process secondary and supplementary alignments
+ for (n = 0; n < bs.n; n++) {
+ if (!(bs.b[n].core.flag & (BAM_FSECONDARY|BAM_FSUPPLEMENTARY))) {
+ // primary
+ continue;
+ }
+
+ // Secondary or supplementary
+ int is_r2 = (bs.b[n].core.flag & BAM_FREAD2) != 0;
+ bam1_t *primary = rnum[is_r2];
+ if (primary) {
+ if (base_mods)
+ fix_MM(primary, &bs.b[n], &state[is_r2]);
+ } else {
+ // Record with base modifications but no known primary
+ //fprintf(stderr, "Unpaired secondary or supplementary\n");
+ if (base_mods)
+ fix_MM(NULL, &bs.b[n], NULL);
+ }
+ }
+
+ // Finally having curated everything, write out all records in their
+ // original ordering
+ for (n = 0; n < bs.n; n++) {
+ bam1_t *cur = &bs.b[n];
+ // We may remove unmapped and secondary alignments
+ if (remove_reads && (cur->core.flag & (BAM_FSECONDARY|BAM_FUNMAP)))
+ continue;
+
+ if (sam_write1(out, header, cur) < 0)
+ goto write_fail;
+ }
}
+ if (result < -1)
+ goto read_fail;
+
sam_hdr_destroy(header);
- bam_destroy1(b[0]);
- bam_destroy1(b[1]);
+
+ for (n = 0; n < bs.ba; n++)
+ free(bs.b[n].data);
+ free(bs.b);
ks_free(&str);
return 0;
print_error_errno("fixmate", "Couldn't write to output file");
fail:
sam_hdr_destroy(header);
- bam_destroy1(b[0]);
- bam_destroy1(b[1]);
+ for (n = 0; n < bs.ba; n++)
+ free(bs.b[n].data);
+ free(bs.b);
ks_free(&str);
return 1;
}
" -u Uncompressed output\n"
" -z, --sanitize FLAG[,FLAG]\n"
" Sanitize alignment fields [defaults to all types]\n"
+" -M Fix base modification tags (MM/ML/MN)\n"
" --no-PG do not add a PG line\n");
sam_global_opt_help(where, "-.O..@-.");
htsThreadPool p = {NULL, 0};
samFile *in = NULL, *out = NULL;
int c, remove_reads = 0, proper_pair_check = 1, add_ct = 0, res = 1,
- mate_score = 0, no_pg = 0, sanitize_flags = FIX_ALL;
+ mate_score = 0, no_pg = 0, sanitize_flags = FIX_ALL, base_mods = 0;
sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
char wmode[4] = {'w', 'b', 0, 0};
static const struct option lopts[] = {
// parse args
if (argc == 1) { usage(stdout); return 0; }
- while ((c = getopt_long(argc, argv, "rpcmO:@:uz:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "rpcmMO:@:uz:", lopts, NULL)) >= 0) {
switch (c) {
case 'r': remove_reads = 1; break;
case 'p': proper_pair_check = 0; break;
case 'c': add_ct = 1; break;
case 'm': mate_score = 1; break;
+ case 'M': base_mods = 1; break;
case 'u': wmode[2] = '0'; break;
case 1: no_pg = 1; break;
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
// run
res = bam_mating_core(in, out, remove_reads, proper_pair_check, add_ct,
- mate_score, arg_list, no_pg, sanitize_flags);
+ mate_score, arg_list, no_pg, sanitize_flags,
+ base_mods);
// cleanup
sam_close(in);
/* bam_mate.c -- fix mate pairing information and clean up flags.
- Copyright (C) 2009, 2011-2017, 2019, 2022 Genome Research Ltd.
+ Copyright (C) 2009, 2011-2017, 2019, 2022, 2024 Genome Research Ltd.
Portions copyright (C) 2011 Broad Institute.
Portions copyright (C) 2012 Peter Cock, The James Hutton Institute.
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
+#include <ctype.h>
#include "htslib/thread_pool.h"
#include "sam_opts.h"
#include "htslib/kstring.h"
* single Reads:
* -if pos == 0 (1 based), tid == -1, or UNMAPPED then set UNMAPPED, pos = 0,
* tid = -1
- * -clear bad flags (PAIRED, MREVERSE, PROPER_PAIR)
+ * -clear bad flags (MREVERSE, PROPER_PAIR)
* -set mpos = 0 (1 based), mtid = -1 and isize = 0
* -write to output
* Paired Reads:
return 0;
}
+// Look for 3 tags in one pass, for efficiencies sake.
+// We also convert the draft tags Mm and Ml to MM and ML here.
+static inline void find_tags(bam1_t *b,
+ char *t1, uint8_t **t1p,
+ char *t2, uint8_t **t2p,
+ char *t3, uint8_t **t3p) {
+ *t1p = *t2p = *t3p = NULL;
+ uint8_t *aux = bam_aux_first(b);
+
+ while (aux) {
+ if (aux[-2] == t1[0] && toupper(aux[-1]) == t1[1]) {
+ *t1p = aux;
+ if (islower(aux[-1]))
+ aux[-1] = t1[1];
+ } else if (aux[-2] == t2[0] && toupper(aux[-1]) == t2[1]) {
+ *t2p = aux;
+ if (islower(aux[-1]))
+ aux[-1] = t2[1];
+ } else if (aux[-2] == t3[0] && toupper(aux[-1]) == t3[1]) {
+ *t3p = aux;
+ if (islower(aux[-1]))
+ aux[-1] = t3[1];
+ }
+ aux = bam_aux_next(b, aux);
+ }
+}
+
+// Return 5' and 3' CIGAR hard-clip counts
+static inline void hard_clips(bam1_t *b, int *end5, int *end3) {
+ uint32_t *cigar = bam_get_cigar(b);
+ int ncigar = b->core.n_cigar;
+ int endL = 0, endR = 0, nh = 0;
+
+ if (ncigar && bam_cigar_op(cigar[0]) == BAM_CHARD_CLIP)
+ endL = bam_cigar_oplen(cigar[0]), nh=1;
+ if (ncigar > nh && bam_cigar_op(cigar[ncigar-1]) == BAM_CHARD_CLIP)
+ endR = bam_cigar_oplen(cigar[ncigar-1]);
+
+ if (b->core.flag & BAM_FREVERSE) {
+ *end5 = endR;
+ *end3 = endL;
+ } else {
+ *end5 = endL;
+ *end3 = endR;
+ }
+}
+
+// Get MM, ML and MN tags, and 5' and 3' hard-clip lengths.
+// MNi is integer copy of MN, or -1 if absent/invalid
+void get_mod_info(bam1_t *b, uint8_t **MM, uint8_t **ML, uint8_t **MN,
+ int *MNi, int *end5, int *end3) {
+ find_tags(b, "MM", MM, "ML", ML, "MN", MN);
+ if (*MN) {
+ int save_errno = errno;
+ errno = 0;
+ *MNi = bam_aux2i(*MN);
+ if (errno == EINVAL)
+ *MNi = -1;
+ errno = save_errno;
+ } else {
+ *MNi = -1;
+ }
+
+ if (*MM)
+ hard_clips(b, end5, end3);
+ else
+ *end5 = *end3 = 0; // don't need if MM not found
+}
+
+typedef struct MM_state {
+ // tags found on "pre" BAM
+ uint8_t *MM, *ML, *MN;
+} MM_state;
+
+uint8_t *MN_enc(uint8_t *tag, uint32_t n) {
+ if (n > UINT16_MAX) {
+ tag[0] = 'I';
+ i32_to_le(n, tag+1);
+ tag += 5;
+ } else if (n > UINT8_MAX) {
+ tag[0] = 'S';
+ i16_to_le(n, tag+1);
+ tag += 3;
+ } else {
+ *tag++ = 'C';
+ *tag++ = n;
+ }
+
+ return tag;
+}
+
+// Trim 5'/3' bases off MM and ML tags, using a previous sequence as a guide.
+int trim_MM(bam1_t *pre, bam1_t *cur, int end5, int end3,
+ uint8_t *MM, uint8_t *ML, uint8_t *MN) {
+ // Count number of bases
+ int counts5[16] = {0}, counts3[16] = {0};
+
+ uint8_t *seq = bam_get_seq(pre);
+ int i;
+ for (i = 0; i < end5; i++)
+ counts5[bam_seqi(seq, i)]++;
+ memcpy(counts3, counts5, 16 * sizeof(*counts3));
+ for (; i < pre->core.l_qseq - end3; i++)
+ counts3[bam_seqi(seq, i)]++;
+
+ // "p" is position in pre.
+ // "q" is position in cur.
+ // Hence move up "p" to start and copy from there to "q".
+ uint8_t *MMp, *MLp, *MMq = NULL, *MLq = NULL;
+ if (ML && ML[0] == 'B' && ML[1] == 'C') {
+ MLp = ML+6;
+ } else {
+ ML = MLp = NULL;
+ }
+ MMq = MM+1;
+ MLq = MLp;
+ for (MMp = MM+1; *MMp; ) {
+ int fundamental = seq_nt16_table[*MMp];
+ while (*MMp && *MMp != ',')
+ *MMq++ = *MMp++;
+ if (*MMp)
+ *MMq++ = *MMp++;
+
+ // Now on comma separated list for MM and BC array for ML. Skip
+ int n = 0;
+ while (*MMp != ';' && n < counts5[fundamental]) {
+ char *endptr;
+ long delta = strtol((char *)MMp, &endptr, 10);
+ if (counts5[fundamental] - n > delta) {
+ // Skip entire delta in MM and ML.
+ // Eg counts[]=10, MM=3,10 ML=<10><20> => MM=10 ML=<20>
+ n += delta+1;
+ if(ML) MLp++;
+ } else if (counts3[fundamental] > counts5[fundamental]) {
+ // Shrink delta, writing MM and ML is unchanged.
+ // Eg counts[]=3, MM=10,4 ML=<10><20> => MM=7,4 ML=<10><20>
+ char num[50];
+ int l = sprintf(num, "%ld",
+ delta - (counts5[fundamental]-n));
+ memcpy((char *)MMq, num, l);
+ MMq += l;
+ *MMq++ = *endptr;
+ n += delta+1;
+ if (ML)
+ *MLq++ = *MLp++;
+ } else {
+ // next base mod is on boundary of 3' clip point
+ break;
+ }
+
+ MMp = (uint8_t *)endptr;
+ if (*MMp != ',')
+ // error? if not ; also?
+ break;
+ MMp++;
+ }
+
+ // Copy
+ while (*MMp != ';' && n < counts3[fundamental]) {
+ char *endptr;
+ long delta = strtol((char *)MMp, &endptr, 10);
+ if (counts3[fundamental] - n > delta) {
+ // Copy entire delta in MM and ML including [,;]
+ memmove(MMq, MMp, (uint8_t *)endptr - MMp + 1);
+ MMq += (uint8_t *)endptr - MMp + 1;
+ n += delta+1;
+ if (ML)
+ *MLq++ = *MLp++;
+ } else {
+ // Next mod is into 3' cutoff, so can terminate MM/ML now
+ n = counts3[fundamental];
+ if (ML)
+ MLp++;
+ }
+
+ MMp = (uint8_t *)endptr;
+ if (*MMp != ',')
+ break;
+ MMp++;
+ }
+
+ // Skip
+ while (*MMp && *MMp != ';') {
+ while (*MMp && *MMp != ',' && *MMp != ';')
+ MMp++;
+ if (*MMp == ',')
+ MMp++;
+
+ if (ML)
+ MLp++;
+ }
+ MMq[-1] = ';'; // replaces , with ; if clipping right
+ if (*MMp)
+ MMp++;
+ }
+
+ MMp++; // skip nul
+ *MMq++ = 0;
+
+ // Adjust ML B array length
+ if (ML)
+ u32_to_le(MLq-(ML+6), ML+2);
+
+ // Move MM and ML down to include their MM:Z and ML:B bits
+ if (MM) MM-=2;
+ if (ML) ML-=2;
+
+ // Now MM/ML are start of tags, MMq/MLq are ends of edited tags,
+ // and MMp/MLp are ends of original tags. Walk through tags taking up
+ // any gaps
+ //
+ // Eg XXXXXXmmmmm--YYYlllll-ZZ (m and l are edited MM and ML tags)
+ // => XXXXXXmmmmmYYYlllllZZ
+
+ uint8_t *tag = bam_get_aux(cur), *tag_end = cur->data + cur->l_data;
+ uint8_t *to = tag;
+ while (tag && tag < tag_end) {
+ if (tag[0] == 'M' && (tag[1] == 'M' || tag[1] == 'm')) {
+ // Slow but easy
+ memmove(to, MM, MMq-MM); // length of new tag
+ to += MMq-MM;
+ tag = MMp; // size of old tag
+ } else if (tag[0] == 'M' && (tag[1] == 'L' || tag[1] == 'l')) {
+ memmove(to, ML, MLq-ML);
+ to += MLq-ML;
+ tag = MLp;
+ } else if (tag[0] == 'M' && tag[1] == 'N') {
+ tag = bam_aux_next(cur, tag+2);
+ // Skip it as we'll overwrite this later, although this
+ // does change the tag order. Instead we could do:
+ //
+ // *to++ = 'M';
+ // *to++ = 'N';
+ // to = MN_enc(to, cur->core.l_qseq);
+ } else {
+ // Want aux_skip, but it's private.
+ // So we use bam_aux_next with work-arounds. :(
+ uint8_t *from = tag;
+ tag = bam_aux_next(cur, tag+2);
+ tag = tag ? tag-2 : tag_end;
+ memmove(to, from, tag-from);
+ to += tag-from;
+ }
+ }
+ cur->l_data = to - cur->data;
+
+ return 0;
+}
+
+// Removes base modification tags: MM, ML and MN.
+// This is more efficient than a series of bam_aux_remove and
+// bam_aux_find calls, as the previous removes shuffle the tags we've
+// previously found. However it's still not optimal.
+void delete_mod_tags(bam1_t *b) {
+ uint8_t *tag = bam_aux_first(b), *next;
+ uint8_t *to = tag;
+ while (tag) {
+ next = bam_aux_next(b, tag);
+ if (tag[-2] == 'M' &&
+ (tag[-1] == 'M' || tag[-1] == 'm' ||
+ tag[-1] == 'L' || tag[-1] == 'l' ||
+ tag[-1] == 'N')) {
+ // Skip. Equivalent to bam_aux_remove without multiple passes
+ } else {
+ // Copy. All these +/-2s are an annoyance caused by the
+ // tag iterator pointing to the byte after the 2-letter code
+ uint8_t *end = next ? next : b->data + b->l_data + 2;
+ if (tag != to)
+ memmove(to-2, tag-2, end-tag);
+ to += end-tag;
+ }
+ tag = next;
+ }
+
+ b->l_data = (to-2) - b->data;
+}
+
+int validate_MM(bam1_t *b, hts_base_mod_state *state) {
+ hts_base_mod mods[10];
+ int n, pos;
+ while ((n = bam_next_basemod(b, state, mods, 10, &pos)) > 0) {
+ // bam_next_basemod will trigger MM out-of-bound checks
+ }
+ return n;
+}
+
+// Fix base modification tags MM, ML and MN.
+// For supplementary-style alignments we may have hard-clipped the sequence
+// and just duplicated the MM/ML tags. Use the primary alignment to get the
+// clipped sequence so we can trim MM/ML accordingly.
+//
+// We call this first on primary reads with pre == NULL. This caches
+// MM and ML data into MM_state.
+//
+// We then call it again on secondary and/or supplementary data with
+// pre == the primary record and pass in the associated state. This then
+// validates MM/ML/MN match, and if not adjusts them if they have hard-clips
+// which yields consistent data.
+//
+// TODO: add sanity check on counts of base types and MM tag to ensure it's
+// possible. We can do this post-trimming, so we sanitize everything.
+//
+// Returns 0 on success,
+// -1 on failure
+int fix_MM(bam1_t *pre, bam1_t *cur, MM_state *state) {
+ int end5, end3;
+ int MNi = 0; // MN of -1 is used as indicator for no valid mods
+
+ if (!pre && state) {
+ // First time we've see this name.
+ // Look for base modification tags and sanity check.
+ get_mod_info(cur, &state->MM, &state->ML, &state->MN, &MNi,
+ &end5, &end3);
+ if (!state->MM) {
+ delete_mod_tags(cur);
+ return 0;
+ }
+
+ if (!end5 && !end3 && MNi <= 0) {
+ // No MN tag, but also no clipping. Assume MM is valid
+ if (cur->core.l_qseq)
+ if (bam_aux_update_int(cur, "MN", cur->core.l_qseq) < 0)
+ return -1;
+ } else if ((end3 || end5) && cur->core.l_qseq != MNi) {
+ // We have hard clips and MN tag, but the MN tag doesn't match
+ // observed sequence length so it appears the hard-clipping
+ // happened after base-mods called without updating.
+ // Fail as this is a primary read.
+ delete_mod_tags(cur);
+ }
+ // Otherwise we assume the base modifications are correct
+
+ } else if (state) {
+ // A supplementary or secondary alignment with known primary
+ uint8_t *cur_MM = NULL, *cur_ML = NULL, *cur_MN = NULL;
+ MNi = -1;
+ get_mod_info(cur, &cur_MM, &cur_ML, &cur_MN, &MNi, &end5, &end3);
+
+ if (!cur_MM) {
+ delete_mod_tags(cur);
+ return 0;
+ }
+
+ // Does MN match seq length? If so, we believe it's already valid
+ if (MNi == cur->core.l_qseq)
+ goto validate;
+
+ // Length mismatch and/or no known length, so check vs full seq.
+ if (pre->core.l_qseq != cur->core.l_qseq + end3 + end5) {
+ delete_mod_tags(cur);
+ return 0;
+ } else if (end5 || end3) {
+ if (MNi < 0 || MNi == pre->core.l_qseq)
+ trim_MM(pre, cur, end5, end3, cur_MM, cur_ML, cur_MN);
+ } // else no hard clips so MM is already valid
+
+ // Set MN so we've validated it, provided seq isn't "*".
+ // inefficient, but minimal compared to everything else
+ if (cur->core.l_qseq)
+ if (bam_aux_update_int(cur, "MN", cur->core.l_qseq) < 0)
+ return -1;
+ }
+
+ validate:
+ ;
+
+ // Also validate MM length matches sequence length. This mirrors the
+ // logic in htslib/sam_mods.c.
+ // For now we take the inefficient approach of using bam_parse_basemod2.
+ // Inefficient, but robust.
+ hts_base_mod_state *mst = hts_base_mod_state_alloc();
+ if (!mst)
+ return -1;
+
+ enum htsLogLevel lvl = hts_get_log_level();
+ hts_set_log_level(HTS_LOG_OFF);
+ if (bam_parse_basemod(cur, mst) < 0)
+ // Maybe we want hts_log_warning still though?
+ delete_mod_tags(cur);
+ if (validate_MM(cur, mst) < 0)
+ delete_mod_tags(cur);
+ hts_set_log_level(lvl);
+ hts_base_mod_state_free(mst);
+
+ return 0;
+}
+
+// Ensure the b[] array is at least n.
+// Returns 0 on success,
+// -1 on failure
+static int grow_b_array(bam1_t **b, int *ba, int n) {
+ if (n < *ba)
+ return 0;
+
+ bam1_t *bnew = realloc(*b, (n+=10) * sizeof(**b));
+ if (!bnew)
+ return -1;
+ *b = bnew;
+
+ // bam_init1 equivalent
+ int i;
+ for (i = *ba; i < n; i++)
+ memset(&(*b)[i], 0, sizeof(bam1_t));
+
+ *b = bnew;
+ *ba = n;
+
+ return 0;
+}
+
+// We have b[0]..b[bn-1] entries all from the same template (qname)
+typedef struct {
+ bam1_t *b;
+ int n, ba; // number used and number allocated
+ int b_next; // b[b_next] for start of next set, -1 if unset
+ int eof; // marker for having seen eof
+} bam_set;
+
+// Fetches a new batch of BAM records all containing the same name.
+// NB: we cache the last (non-matching) name in b[n], so we can use it to
+// start the next batch.
+// Returns the number of records on success,
+// <0 on failure or EOF (sam_read1 return vals)
+static int next_template(samFile *in, sam_hdr_t *header, bam_set *bs,
+ int sanitize_flags) {
+ int result;
+
+ if (bs->eof)
+ return -1;
+
+ // First time through, prime the template name
+ if (bs->b_next < 0) {
+ if (grow_b_array(&bs->b, &bs->ba, 1) < 0)
+ return -2;
+ result = sam_read1(in, header, &bs->b[0]);
+ if (result < 0)
+ return result;
+ if (bam_sanitize(header, &bs->b[0], sanitize_flags) < 0)
+ return -2;
+ } else {
+ // Otherwise use the previous template name read
+ bam1_t btmp = bs->b[0];
+ bs->b[0] = bs->b[bs->b_next];
+ bs->b[bs->b_next] = btmp; // For ->{,l_,m_}data
+ }
+ bs->n = 1;
+
+ // Now keep reading until we find a read that mismatches or we hit eof.
+ char *name = bam_get_qname(&bs->b[0]);
+ for (;;) {
+ if (grow_b_array(&bs->b, &bs->ba, bs->n+1) < 0)
+ return -2;
+
+ result = sam_read1(in, header, &bs->b[bs->n]);
+ if (result < -1)
+ return result;
+
+ if (result < 0) {
+ bs->eof = 1;
+ bs->b_next = -1;
+ break;
+ } else {
+ if (bam_sanitize(header, &bs->b[bs->n], sanitize_flags) < 0)
+ return -2;
+
+ bs->b_next = bs->n;
+ if (strcmp(name, bam_get_qname(&bs->b[bs->n])) != 0)
+ break;
+ }
+
+ bs->n++;
+ }
+
+ return bs->n;
+}
+
// currently, this function ONLY works if each read has one hit
+//
+// Returns 0 on success,
+// >0 on failure
static int bam_mating_core(samFile *in, samFile *out, int remove_reads,
int proper_pair_check, int add_ct,
int do_mate_scoring, char *arg_list, int no_pg,
- int sanitize_flags)
+ int sanitize_flags, int base_mods)
{
sam_hdr_t *header;
- bam1_t *b[2] = { NULL, NULL };
- int curr, has_prev, result;
- hts_pos_t pre_end = 0, cur_end = 0;
+ int result, n;
kstring_t str = KS_INITIALIZE;
+ bam_set bs = {NULL, 0, 0, -1, 0};
header = sam_hdr_read(in);
if (header == NULL) {
if (sam_hdr_write(out, header) < 0) goto write_fail;
- b[0] = bam_init1();
- b[1] = bam_init1();
- curr = 0; has_prev = 0;
- while ((result = sam_read1(in, header, b[curr])) >= 0) {
- bam1_t *cur = b[curr], *pre = b[1-curr];
- if (bam_sanitize(header, cur, sanitize_flags) < 0)
- goto fail;
- if (cur->core.flag & BAM_FSECONDARY)
- {
- if ( !remove_reads ) {
- if (sam_write1(out, header, cur) < 0) goto write_fail;
+ // Iterate template by template fetching bs->n records at a time
+ while ((result = next_template(in, header, &bs, sanitize_flags)) >= 0) {
+ bam1_t *cur = NULL, *pre = NULL, *rnum[2] = {NULL, NULL};
+ int prev = -1, curr = -1;
+ hts_pos_t pre_end = 0, cur_end = 0;
+
+ // Find and fix up primary alignments
+ MM_state state[2];
+ for (n = 0; n < bs.n; n++) {
+ int is_r2 = (bs.b[n].core.flag & BAM_FREAD2) != 0;
+ if (bs.b[n].core.flag & (BAM_FSECONDARY | BAM_FSUPPLEMENTARY))
+ continue;
+
+ if (base_mods)
+ if (fix_MM(NULL, &bs.b[n], &state[is_r2]) < 0)
+ goto fail;
+
+ if (!pre) {
+ pre = &bs.b[prev = n];
+ rnum[(pre->core.flag & BAM_FREAD2) != 0] = pre;
+
+ pre_end = (pre->core.flag & BAM_FUNMAP) == 0
+ ? bam_endpos(pre) : 0;
+ continue;
}
- continue; // skip secondary alignments
- }
- if (cur->core.flag & BAM_FSUPPLEMENTARY)
- {
- if (sam_write1(out, header, cur) < 0) goto write_fail;
- continue; // pass supplementary alignments through unchanged (TODO:make them match read they came from)
- }
- if ((cur->core.flag&BAM_FUNMAP) == 0) // If mapped calculate end
- {
- cur_end = bam_endpos(cur);
- }
- if (has_prev) { // do we have a pair of reads to examine?
- if (strcmp(bam_get_qname(cur), bam_get_qname(pre)) == 0) { // identical pair name
- pre->core.flag |= BAM_FPAIRED;
- cur->core.flag |= BAM_FPAIRED;
- if (sync_mate(pre, cur)) goto fail;
-
- if (pre->core.tid == cur->core.tid && !(cur->core.flag&(BAM_FUNMAP|BAM_FMUNMAP))
- && !(pre->core.flag&(BAM_FUNMAP|BAM_FMUNMAP))) // if safe set TLEN/ISIZE
- {
- hts_pos_t cur5, pre5;
- cur5 = (cur->core.flag&BAM_FREVERSE)? cur_end : cur->core.pos;
- pre5 = (pre->core.flag&BAM_FREVERSE)? pre_end : pre->core.pos;
- cur->core.isize = pre5 - cur5; pre->core.isize = cur5 - pre5;
- } else cur->core.isize = pre->core.isize = 0;
- if (add_ct) bam_template_cigar(pre, cur, &str);
- // TODO: Add code to properly check if read is in a proper pair based on ISIZE distribution
- if (proper_pair_check && !plausibly_properly_paired(pre,cur)) {
- pre->core.flag &= ~BAM_FPROPER_PAIR;
- cur->core.flag &= ~BAM_FPROPER_PAIR;
- }
+ // Note, more than 2 primary alignments will use 'curr' as last
+ cur = &bs.b[curr = n];
+ rnum[(cur->core.flag & BAM_FREAD2) != 0] = cur;
+ cur_end = (cur->core.flag & BAM_FUNMAP) == 0
+ ? bam_endpos(cur) : 0;
+
+ pre->core.flag |= BAM_FPAIRED;
+ cur->core.flag |= BAM_FPAIRED;
+ if (sync_mate(pre, cur))
+ goto fail;
+
+ // If safe set TLEN/ISIZE
+ if (pre->core.tid == cur->core.tid
+ && !(cur->core.flag & (BAM_FUNMAP | BAM_FMUNMAP))
+ && !(pre->core.flag & (BAM_FUNMAP | BAM_FMUNMAP))) {
+ hts_pos_t cur5, pre5;
+ cur5 = (cur->core.flag & BAM_FREVERSE)
+ ? cur_end
+ : cur->core.pos;
+ pre5 = (pre->core.flag & BAM_FREVERSE)
+ ? pre_end
+ : pre->core.pos;
+ cur->core.isize = pre5 - cur5;
+ pre->core.isize = cur5 - pre5;
+ } else {
+ cur->core.isize = pre->core.isize = 0;
+ }
- if (do_mate_scoring) {
- if ((add_mate_score(pre, cur) == -1) || (add_mate_score(cur, pre) == -1)) {
- fprintf(samtools_stderr, "[bam_mating_core] ERROR: unable to add mate score.\n");
- goto fail;
- }
- }
+ if (add_ct)
+ bam_template_cigar(pre, cur, &str);
- // Write out result
- if ( !remove_reads ) {
- if (sam_write1(out, header, pre) < 0) goto write_fail;
- if (sam_write1(out, header, cur) < 0) goto write_fail;
- } else {
- // If we have to remove reads make sure we do it in a way that doesn't create orphans with bad flags
- if(pre->core.flag&BAM_FUNMAP) cur->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if(cur->core.flag&BAM_FUNMAP) pre->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if(!(pre->core.flag&BAM_FUNMAP)) {
- if (sam_write1(out, header, pre) < 0) goto write_fail;
- }
- if(!(cur->core.flag&BAM_FUNMAP)) {
- if (sam_write1(out, header, cur) < 0) goto write_fail;
- }
- }
- has_prev = 0;
- } else { // unpaired? clear bad info and write it out
- pre->core.mtid = -1; pre->core.mpos = -1; pre->core.isize = 0;
- pre->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if ( !remove_reads || !(pre->core.flag&BAM_FUNMAP) ) {
- if (sam_write1(out, header, pre) < 0) goto write_fail;
+ // TODO: Add code to properly check if read is in a proper
+ // pair based on ISIZE distribution
+ if (proper_pair_check && !plausibly_properly_paired(pre,cur)) {
+ pre->core.flag &= ~BAM_FPROPER_PAIR;
+ cur->core.flag &= ~BAM_FPROPER_PAIR;
+ }
+
+ if (do_mate_scoring) {
+ if ((add_mate_score(pre, cur) == -1) ||
+ (add_mate_score(cur, pre) == -1)) {
+ fprintf(samtools_stderr, "[bam_mating_core] ERROR: "
+ "unable to add mate score.\n");
+ goto fail;
}
}
- } else has_prev = 1;
- curr = 1 - curr;
- pre_end = cur_end;
- }
- if (result < -1) goto read_fail;
- if (has_prev && !remove_reads) { // If we still have a BAM in the buffer it must be unpaired
- bam1_t *pre = b[1-curr];
- if (pre->core.tid < 0 || pre->core.pos < 0 || pre->core.flag&BAM_FUNMAP) { // If unmapped
- pre->core.flag |= BAM_FUNMAP;
- pre->core.tid = -1;
- pre->core.pos = -1;
+
+ // If we have to remove reads make sure we do it in a way that
+ // doesn't create orphans with bad flags
+ if (remove_reads) {
+ if (pre->core.flag&BAM_FUNMAP)
+ cur->core.flag &=
+ ~(BAM_FMREVERSE|BAM_FPROPER_PAIR);
+ if (cur->core.flag&BAM_FUNMAP)
+ pre->core.flag &=
+ ~(BAM_FMREVERSE|BAM_FPROPER_PAIR);
+ }
}
- pre->core.mtid = -1; pre->core.mpos = -1; pre->core.isize = 0;
- pre->core.flag &= ~(BAM_FPAIRED|BAM_FMREVERSE|BAM_FPROPER_PAIR);
- if (sam_write1(out, header, pre) < 0) goto write_fail;
+ // Handle unpaired primary data
+ if (!cur && pre) {
+ pre->core.mtid = -1;
+ pre->core.mpos = -1;
+ pre->core.isize = 0;
+ pre->core.flag &= ~(BAM_FMREVERSE|BAM_FPROPER_PAIR);
+ }
+
+ // Now process secondary and supplementary alignments
+ for (n = 0; n < bs.n; n++) {
+ if (!(bs.b[n].core.flag & (BAM_FSECONDARY|BAM_FSUPPLEMENTARY))) {
+ // primary
+ continue;
+ }
+
+ // Secondary or supplementary
+ int is_r2 = (bs.b[n].core.flag & BAM_FREAD2) != 0;
+ bam1_t *primary = rnum[is_r2];
+ if (primary) {
+ if (base_mods)
+ fix_MM(primary, &bs.b[n], &state[is_r2]);
+ } else {
+ // Record with base modifications but no known primary
+ //fprintf(samtools_stderr, "Unpaired secondary or supplementary\n");
+ if (base_mods)
+ fix_MM(NULL, &bs.b[n], NULL);
+ }
+ }
+
+ // Finally having curated everything, write out all records in their
+ // original ordering
+ for (n = 0; n < bs.n; n++) {
+ bam1_t *cur = &bs.b[n];
+ // We may remove unmapped and secondary alignments
+ if (remove_reads && (cur->core.flag & (BAM_FSECONDARY|BAM_FUNMAP)))
+ continue;
+
+ if (sam_write1(out, header, cur) < 0)
+ goto write_fail;
+ }
}
+ if (result < -1)
+ goto read_fail;
+
sam_hdr_destroy(header);
- bam_destroy1(b[0]);
- bam_destroy1(b[1]);
+
+ for (n = 0; n < bs.ba; n++)
+ free(bs.b[n].data);
+ free(bs.b);
ks_free(&str);
return 0;
print_error_errno("fixmate", "Couldn't write to output file");
fail:
sam_hdr_destroy(header);
- bam_destroy1(b[0]);
- bam_destroy1(b[1]);
+ for (n = 0; n < bs.ba; n++)
+ free(bs.b[n].data);
+ free(bs.b);
ks_free(&str);
return 1;
}
" -u Uncompressed output\n"
" -z, --sanitize FLAG[,FLAG]\n"
" Sanitize alignment fields [defaults to all types]\n"
+" -M Fix base modification tags (MM/ML/MN)\n"
" --no-PG do not add a PG line\n");
sam_global_opt_help(where, "-.O..@-.");
htsThreadPool p = {NULL, 0};
samFile *in = NULL, *out = NULL;
int c, remove_reads = 0, proper_pair_check = 1, add_ct = 0, res = 1,
- mate_score = 0, no_pg = 0, sanitize_flags = FIX_ALL;
+ mate_score = 0, no_pg = 0, sanitize_flags = FIX_ALL, base_mods = 0;
sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
char wmode[4] = {'w', 'b', 0, 0};
static const struct option lopts[] = {
// parse args
if (argc == 1) { usage(samtools_stdout); return 0; }
- while ((c = getopt_long(argc, argv, "rpcmO:@:uz:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "rpcmMO:@:uz:", lopts, NULL)) >= 0) {
switch (c) {
case 'r': remove_reads = 1; break;
case 'p': proper_pair_check = 0; break;
case 'c': add_ct = 1; break;
case 'm': mate_score = 1; break;
+ case 'M': base_mods = 1; break;
case 'u': wmode[2] = '0'; break;
case 1: no_pg = 1; break;
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
// run
res = bam_mating_core(in, out, remove_reads, proper_pair_check, add_ct,
- mate_score, arg_list, no_pg, sanitize_flags);
+ mate_score, arg_list, no_pg, sanitize_flags,
+ base_mods);
// cleanup
sam_close(in);
}
}
-int calmd_usage() {
+int calmd_usage(void) {
fprintf(stderr,
"Usage: samtools calmd [-eubrAESQ] <aln.bam> <ref.fasta>\n"
"Options:\n"
}
}
-int calmd_usage() {
+int calmd_usage(void) {
fprintf(samtools_stderr,
"Usage: samtools calmd [-eubrAESQ] <aln.bam> <ref.fasta>\n"
"Options:\n"
int bam_plbuf_push(const bam1_t *b, bam_plbuf_t *buf);
-/* Exported from bam_plcmd.c */
-int pileup_seq(FILE *fp, const bam_pileup1_t *p, hts_pos_t pos,
- hts_pos_t ref_len, const char *ref, kstring_t *ks,
- int rev_del, int no_ins, int no_ins_mods,
- int no_del, int no_ends);
#ifdef __cplusplus
}
#endif
/* bam_plcmd.c -- mpileup subcommand.
- Copyright (C) 2008-2015, 2019-2021 Genome Research Ltd.
+ Copyright (C) 2008-2015, 2019-2021, 2023-2024 Genome Research Ltd.
Portions copyright (C) 2009-2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
#define dummy_free(p)
KLIST_INIT(auxlist, char *, dummy_free)
-static inline int printw(int c, FILE *fp)
-{
- char buf[16];
- int l, x;
- if (c == 0) return fputc('0', fp);
- for (l = 0, x = c < 0? -c : c; x > 0; x /= 10) buf[l++] = x%10 + '0';
- if (c < 0) buf[l++] = '-';
- buf[l] = 0;
- for (x = 0; x < l/2; ++x) {
- int y = buf[x]; buf[x] = buf[l-1-x]; buf[l-1-x] = y;
- }
- fputs(buf, fp);
- return 0;
-}
-
-int pileup_seq(FILE *fp, const bam_pileup1_t *p, hts_pos_t pos,
- hts_pos_t ref_len, const char *ref, kstring_t *ks,
+int pileup_seq(kstring_t *ks_seq, const bam_pileup1_t *p, hts_pos_t pos,
+ hts_pos_t ref_len, const char *ref, kstring_t *ks_mod,
int rev_del, int no_ins, int no_ins_mods,
int no_del, int no_ends)
{
no_ins_mods |= no_ins;
- int j;
+ int j, err = 0;
hts_base_mod_state *m = p->cd.p;
if (!no_ends && p->is_head) {
- putc('^', fp);
- putc(p->b->core.qual > 93? 126 : p->b->core.qual + 33, fp);
+ err |= kputc_('^', ks_seq) < 0;
+ err |= kputc_(p->b->core.qual > 93 ? 126 : p->b->core.qual + 33,
+ ks_seq) < 0;
}
if (!p->is_del) {
+ // See seq_nt16_str in htslib/hts.c
+ const char seq_nt_str_lc[] = ",acmgrsvtwyhkdbn"; // reverse strand
+ const char seq_nt_str_uc[] = ".ACMGRSVTWYHKDBN";
int c = p->qpos < p->b->core.l_qseq
- ? seq_nt16_str[bam_seqi(bam_get_seq(p->b), p->qpos)]
- : 'N';
+ ? bam_seqi(bam_get_seq(p->b), p->qpos)
+ : 15 /*N*/;
if (ref) {
- int rb = pos < ref_len? ref[pos] : 'N';
- if (c == '=' || seq_nt16_table[c] == seq_nt16_table[rb]) c = bam_is_rev(p->b)? ',' : '.';
- else c = bam_is_rev(p->b)? tolower(c) : toupper(c);
- } else {
- if (c == '=') c = bam_is_rev(p->b)? ',' : '.';
- else c = bam_is_rev(p->b)? tolower(c) : toupper(c);
+ int rb = pos < ref_len
+ ? seq_nt16_table[(uint8_t)(ref[pos])]
+ : 15/*N*/;
+ if (c == rb)
+ c = 0; // "=", which becomes . or ,
}
- putc(c, fp);
+ c = bam_is_rev(p->b)
+ ? seq_nt_str_lc[c]
+ : seq_nt_str_uc[c];
+ err |= kputc_(c, ks_seq) < 0;
+
if (m) {
int nm;
hts_base_mod mod[256];
if ((nm = bam_mods_at_qpos(p->b, p->qpos, m, mod, 256)) > 0) {
- putc('[', fp);
+ err |= kputc_('[', ks_seq) < 0;
int j;
for (j = 0; j < nm && j < 256; j++) {
char qual[20];
*qual = 0;
if (mod[j].modified_base < 0)
// ChEBI
- fprintf(fp, "%c(%d)%s", "+-"[mod[j].strand],
- -mod[j].modified_base, qual);
+ err |= ksprintf(ks_seq, "%c(%d)%s",
+ "+-"[mod[j].strand],
+ -mod[j].modified_base, qual) < 0;
else
- fprintf(fp, "%c%c%s", "+-"[mod[j].strand],
- mod[j].modified_base, qual);
+ err |= ksprintf(ks_seq, "%c%c%s", "+-"[mod[j].strand],
+ mod[j].modified_base, qual) < 0;
}
- putc(']', fp);
+ err |= kputc_(']', ks_seq) < 0;
}
}
- } else putc(p->is_refskip? (bam_is_rev(p->b)? '<' : '>') : ((bam_is_rev(p->b) && rev_del) ? '#' : '*'), fp);
+ } else {
+ err |= kputc_(p->is_refskip
+ ? (bam_is_rev(p->b)? '<' : '>')
+ : ((bam_is_rev(p->b) && rev_del) ? '#' : '*'),
+ ks_seq) < 0;
+ }
+
int del_len = -p->indel;
if (p->indel > 0) {
int len = bam_plp_insertion_mod(p, m && !no_ins_mods ? m : NULL,
- ks, &del_len);
+ ks_mod, &del_len);
if (len < 0) {
print_error("mpileup", "bam_plp_insertion() failed");
return -1;
}
if (no_ins < 2) {
- putc('+', fp);
- printw(len, fp);
+ err |= kputc_('+', ks_seq) < 0;
+ err |= kputuw(len, ks_seq) < 0;
}
if (!no_ins) {
+ kstring_t *ks = ks_mod;
if (bam_is_rev(p->b)) {
char pad = rev_del ? '#' : '*';
int in_mod = 0;
for (j = 0; j < ks->l; j++) {
if (ks->s[j] == '[') in_mod = 1;
else if (ks->s[j] == ']') in_mod = 0;
- putc(ks->s[j] != '*'
- ? (in_mod ? ks->s[j] : tolower(ks->s[j]))
- : pad, fp);
+ err |= kputc_(ks->s[j] != '*'
+ ? (in_mod ? ks->s[j] : tolower(ks->s[j]))
+ : pad, ks_seq) < 0;
}
} else {
int in_mod = 0;
for (j = 0; j < ks->l; j++) {
if (ks->s[j] == '[') in_mod = 1;
if (ks->s[j] == ']') in_mod = 0;
- putc(in_mod ? ks->s[j] : toupper(ks->s[j]), fp);
+ err |= kputc_(in_mod ? ks->s[j] : toupper(ks->s[j]),
+ ks_seq) < 0;
}
}
}
}
+
if (del_len > 0) {
if (no_del < 2)
- printw(-del_len, fp);
+ err |= kputw(-del_len, ks_seq) < 0;
if (!no_del) {
for (j = 1; j <= del_len; ++j) {
int c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N';
- putc(bam_is_rev(p->b)? tolower(c) : toupper(c), fp);
+ err |= kputc_(bam_is_rev(p->b)? tolower(c) : toupper(c),
+ ks_seq) < 0;
}
}
}
- if (!no_ends && p->is_tail) putc('$', fp);
- return 0;
+
+ if (!no_ends && p->is_tail)
+ err |= kputc_('$', ks_seq) < 0;
+
+ return -err;
}
#include "sample.h"
#define MPLP_PRINT_MAPQ_CHAR (1<<11)
#define MPLP_PRINT_QPOS (1<<12)
+// Start of struct active_cols elements
#define MPLP_PRINT_QNAME (1<<13)
#define MPLP_PRINT_FLAG (1<<14)
#define MPLP_PRINT_RNAME (1<<15)
#define MPLP_PRINT_TLEN (1<<21)
#define MPLP_PRINT_SEQ (1<<22)
#define MPLP_PRINT_QUAL (1<<23)
-#define MPLP_PRINT_MODS (1<<24)
-#define MPLP_PRINT_QPOS5 (1<<25)
+#define MPLP_PRINT_RLEN (1<<24)
+// Must occur after struct active_cols element list
+#define MPLP_PRINT_MODS (1<<25)
+#define MPLP_PRINT_QPOS5 (1<<26)
-#define MPLP_PRINT_LAST (1<<26) // terminator for loop
+#define MPLP_PRINT_LAST (1<<27) // terminator for loop
#define MPLP_MAX_DEPTH 8000
#define MPLP_MAX_INDEL_DEPTH 250
int supported;
};
- const struct active_cols colnames[11] = {
- {"QNAME", 1}, {"FLAG", 1}, {"RNAME", 1}, {"POS", 1}, {"MAPQ", 1}, {"CIGAR", 0}, {"RNEXT", 1}, {"PNEXT", 1}, {"TLEN", 0}, {"SEQ", 0}, {"QUAL", 0}
+ const struct active_cols colnames[12] = {
+ {"QNAME", 1}, {"FLAG", 1}, {"RNAME", 1}, {"POS", 1}, {"MAPQ", 1},
+ {"CIGAR", 0}, {"RNEXT", 1}, {"PNEXT", 1}, {"TLEN", 0}, {"SEQ", 0},
+ {"QUAL", 0}, {"RLEN", 1},
};
- int i, f = MPLP_PRINT_QNAME, colno = 11;
+ int i, f = MPLP_PRINT_QNAME, colno = sizeof(colnames)/sizeof(*colnames);
for (i = 0; i < colno; i++, f <<= 1)
if (colnames[i].supported)
khash_str2int_set(colhash, colnames[i].name, f);
* @param fn filenames
* @param fn_idx index filenames
*/
-static int mpileup(mplp_conf_t *conf, int n, char **fn, char **fn_idx)
+static int mpileup(mplp_conf_t *conf, int nfn, char **fn, char **fn_idx)
{
mplp_aux_t **data;
int i, tid, *n_plp, tid0 = 0, max_depth;
memset(&gplp, 0, sizeof(mplp_pileup_t));
memset(&buf, 0, sizeof(kstring_t));
- data = calloc(n, sizeof(mplp_aux_t*));
- plp = calloc(n, sizeof(bam_pileup1_t*));
- n_plp = calloc(n, sizeof(int));
+ data = calloc(nfn, sizeof(mplp_aux_t*));
+ plp = calloc(nfn, sizeof(bam_pileup1_t*));
+ n_plp = calloc(nfn, sizeof(int));
sm = bam_smpl_init();
- if (n == 0) {
+ if (nfn == 0) {
fprintf(stderr,"[%s] no input file/data given\n", __func__);
exit(EXIT_FAILURE);
}
// read the header of each file in the list and initialize data
refs_t *refs = NULL;
- for (i = 0; i < n; ++i) {
+ for (i = 0; i < nfn; ++i) {
sam_hdr_t *h_tmp;
data[i] = calloc(1, sizeof(mplp_aux_t));
data[i]->fp = sam_open_format(fn[i], "rb", &conf->ga.in);
data[i]->h = h;
}
}
- fprintf(stderr, "[%s] %d samples in %d input files\n", __func__, sm->n, n);
+ fprintf(stderr, "[%s] %d samples in %d input files\n",
+ __func__, sm->n, nfn);
pileup_fp = conf->output_fname? fopen(conf->output_fname, "w") : stdout;
}
// init pileup
- iter = bam_mplp_init(n, mplp_func, (void**)data);
+ iter = bam_mplp_init(nfn, mplp_func, (void**)data);
if (conf->flag & MPLP_PRINT_MODS) {
bam_mplp_constructor(iter, pileup_cd_create);
bam_mplp_destructor(iter, pileup_cd_destroy);
fprintf(stderr, "[%s] Max depth set to maximum value (%d)\n", __func__, INT_MAX);
} else {
max_depth = conf->max_depth;
- if ( max_depth * n > 1<<20 )
+ if ( max_depth * nfn > 1<<20 )
fprintf(stderr, "[%s] Combined max depth is above 1M. Potential memory hog!\n", __func__);
}
int ret;
int last_tid = -1;
hts_pos_t last_pos = -1;
+ int one_seq = 0;
// begin pileup
+ kstring_t ks_seq = KS_INITIALIZE;
+ kstring_t ks_mod = KS_INITIALIZE;
+ kstring_t ks_qual = KS_INITIALIZE;
while ( (ret=bam_mplp64_auto(iter, &tid, &pos, n_plp, plp)) > 0) {
+ one_seq = 1; // at least 1 output
if (conf->reg && (pos < beg0 || pos >= end0)) continue; // out of the region requested
- mplp_get_ref(data[0], tid, &ref, &ref_len);
- //printf("tid=%d len=%d ref=%p/%s\n", tid, ref_len, ref, ref);
if (conf->all) {
// Deal with missing portions of previous tids
while (tid > last_tid) {
while (++last_pos < sam_hdr_tid2len(h, last_tid)) {
if (conf->bed && bed_overlap(conf->bed, sam_hdr_tid2name(h, last_tid), last_pos, last_pos + 1) == 0)
continue;
- print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, n, ref, ref_len);
+ print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, nfn, ref, ref_len);
}
}
last_tid++;
last_pos = -1;
if (conf->all < 2)
break;
+ if (tid > last_tid)
+ // multiple missing references and -aa used
+ mplp_get_ref(data[0], last_tid, &ref, &ref_len);
}
}
+ mplp_get_ref(data[0], tid, &ref, &ref_len);
+
if (conf->all) {
// Deal with missing portion of current tid
while (++last_pos < pos) {
if (conf->reg && last_pos < beg0) continue; // out of range; skip
if (conf->bed && bed_overlap(conf->bed, sam_hdr_tid2name(h, tid), last_pos, last_pos + 1) == 0)
continue;
- print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, tid), last_pos, n, ref, ref_len);
+ print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, tid), last_pos, nfn, ref, ref_len);
}
last_tid = tid;
last_pos = pos;
if (conf->bed && tid >= 0 && !bed_overlap(conf->bed, sam_hdr_tid2name(h, tid), pos, pos+1)) continue;
fprintf(pileup_fp, "%s\t%"PRIhts_pos"\t%c", sam_hdr_tid2name(h, tid), pos + 1, (ref && pos < ref_len)? ref[pos] : 'N');
- for (i = 0; i < n; ++i) {
- int j, cnt;
+ for (i = 0; i < nfn; ++i) {
+ int j, cnt, err = 0;
+ ks_clear(&ks_seq);
+ ks_clear(&ks_qual);
+ ks_clear(&ks_mod);
for (j = cnt = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = plp[i] + j;
int c = p->qpos < p->b->core.l_qseq
? bam_get_qual(p->b)[p->qpos]
: 0;
- if (c >= conf->min_baseQ) ++cnt;
+ if (c >= conf->min_baseQ) {
+ // Build up seq
+ err |= pileup_seq(&ks_seq, plp[i] + j, pos, ref_len,
+ ref, &ks_mod, conf->rev_del,
+ conf->no_ins, conf->no_ins_mods,
+ conf->no_del, conf->no_ends) < 0;
+
+ // Build up qual
+ err |= kputc_(c+33 < 126 ? c+33 : 126, &ks_qual) < 0;
+ cnt++;
+ }
+ }
+ if (err) {
+ ret = 1;
+ goto fail;
}
fprintf(pileup_fp, "\t%d\t", cnt);
+
if (n_plp[i] == 0) {
fputs("*\t*", pileup_fp);
int flag_value = MPLP_PRINT_MAPQ_CHAR;
fputs("\t*", pileup_fp);
}
} else {
- int n = 0;
- kstring_t ks = KS_INITIALIZE;
- for (j = 0; j < n_plp[i]; ++j) {
- const bam_pileup1_t *p = plp[i] + j;
- int c = p->qpos < p->b->core.l_qseq
- ? bam_get_qual(p->b)[p->qpos]
- : 0;
- if (c >= conf->min_baseQ) {
- n++;
- if (pileup_seq(pileup_fp, plp[i] + j, pos, ref_len,
- ref, &ks, conf->rev_del,
- conf->no_ins, conf->no_ins_mods,
- conf->no_del, conf->no_ends) < 0) {
- ret = 1;
- goto fail;
- }
- }
+ if (ks_seq.l) {
+ fwrite(ks_seq.s, 1, ks_seq.l, pileup_fp);
+ } else {
+ putc('*', pileup_fp);
}
- if (!n) putc('*', pileup_fp);
-
- /* Print base qualities */
- n = 0;
- ks_free(&ks);
putc('\t', pileup_fp);
- for (j = 0; j < n_plp[i]; ++j) {
- const bam_pileup1_t *p = plp[i] + j;
- int c = p->qpos < p->b->core.l_qseq
- ? bam_get_qual(p->b)[p->qpos]
- : 0;
- if (c >= conf->min_baseQ) {
- c = c + 33 < 126? c + 33 : 126;
- putc(c, pileup_fp);
- n++;
- }
+
+ if (ks_qual.l) {
+ fwrite(ks_qual.s, 1, ks_qual.l, pileup_fp);
+ } else {
+ putc('*', pileup_fp);
}
- if (!n) putc('*', pileup_fp);
/* Print selected columns */
int flag_value = MPLP_PRINT_MAPQ_CHAR;
while(flag_value < MPLP_PRINT_LAST) {
if (flag_value != MPLP_PRINT_MODS
&& (conf->flag & flag_value)) {
- n = 0;
+ int n = 0;
putc('\t', pileup_fp);
for (j = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = &plp[i][j];
case MPLP_PRINT_PNEXT:
fprintf(pileup_fp, "%"PRId64, (int64_t) p->b->core.mpos + 1);
break;
+ case MPLP_PRINT_RLEN:
+ fprintf(pileup_fp, "%d", p->b->core.l_qseq);
+ break;
}
}
if (!n) putc('*', pileup_fp);
if (auxlist_p && auxlist_p->size) {
kliter_t(auxlist) *aux;
for (aux = kl_begin(auxlist_p); aux != kl_end(auxlist_p); aux = kl_next(aux)) {
- n = 0;
+ int n = 0; // NB shadows outer loop
putc('\t', pileup_fp);
for (j = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = &plp[i][j];
putc('\n', pileup_fp);
}
+ ks_free(&ks_seq);
+ ks_free(&ks_mod);
+ ks_free(&ks_qual);
+
if (ret < 0) {
print_error("mpileup", "error reading from input file");
ret = EXIT_FAILURE;
last_tid = tid0;
last_pos = beg0-1;
mplp_get_ref(data[0], tid0, &ref, &ref_len);
+ } else if (last_tid < 0 && !one_seq && conf->all > 1) {
+ last_tid = 0; // --aa on a blank file
}
- while (last_tid >= 0 && last_tid < sam_hdr_nref(h)) {
+ while (last_tid >= 0 && last_tid < sam_hdr_nref(h)) {
+ mplp_get_ref(data[0], last_tid, &ref, &ref_len);
while (++last_pos < sam_hdr_tid2len(h, last_tid)) {
if (last_pos >= end0) break;
if (conf->bed && bed_overlap(conf->bed, sam_hdr_tid2name(h, last_tid), last_pos, last_pos + 1) == 0)
continue;
- print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, n, ref, ref_len);
+ print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, nfn, ref, ref_len);
}
last_tid++;
last_pos = -1;
free(gplp.plp); free(gplp.n_plp); free(gplp.m_plp);
bam_mplp_destroy(iter);
sam_hdr_destroy(h);
- for (i = 0; i < n; ++i) {
+ for (i = 0; i < nfn; ++i) {
sam_close(data[i]->fp);
if (data[i]->iter) hts_itr_destroy(data[i]->iter);
free(data[i]);
/* bam_plcmd.c -- mpileup subcommand.
- Copyright (C) 2008-2015, 2019-2021 Genome Research Ltd.
+ Copyright (C) 2008-2015, 2019-2021, 2023-2024 Genome Research Ltd.
Portions copyright (C) 2009-2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
#define dummy_free(p)
KLIST_INIT(auxlist, char *, dummy_free)
-static inline int printw(int c, FILE *fp)
-{
- char buf[16];
- int l, x;
- if (c == 0) return fputc('0', fp);
- for (l = 0, x = c < 0? -c : c; x > 0; x /= 10) buf[l++] = x%10 + '0';
- if (c < 0) buf[l++] = '-';
- buf[l] = 0;
- for (x = 0; x < l/2; ++x) {
- int y = buf[x]; buf[x] = buf[l-1-x]; buf[l-1-x] = y;
- }
- fputs(buf, fp);
- return 0;
-}
-
-int pileup_seq(FILE *fp, const bam_pileup1_t *p, hts_pos_t pos,
- hts_pos_t ref_len, const char *ref, kstring_t *ks,
+int pileup_seq(kstring_t *ks_seq, const bam_pileup1_t *p, hts_pos_t pos,
+ hts_pos_t ref_len, const char *ref, kstring_t *ks_mod,
int rev_del, int no_ins, int no_ins_mods,
int no_del, int no_ends)
{
no_ins_mods |= no_ins;
- int j;
+ int j, err = 0;
hts_base_mod_state *m = p->cd.p;
if (!no_ends && p->is_head) {
- putc('^', fp);
- putc(p->b->core.qual > 93? 126 : p->b->core.qual + 33, fp);
+ err |= kputc_('^', ks_seq) < 0;
+ err |= kputc_(p->b->core.qual > 93 ? 126 : p->b->core.qual + 33,
+ ks_seq) < 0;
}
if (!p->is_del) {
+ // See seq_nt16_str in htslib/hts.c
+ const char seq_nt_str_lc[] = ",acmgrsvtwyhkdbn"; // reverse strand
+ const char seq_nt_str_uc[] = ".ACMGRSVTWYHKDBN";
int c = p->qpos < p->b->core.l_qseq
- ? seq_nt16_str[bam_seqi(bam_get_seq(p->b), p->qpos)]
- : 'N';
+ ? bam_seqi(bam_get_seq(p->b), p->qpos)
+ : 15 /*N*/;
if (ref) {
- int rb = pos < ref_len? ref[pos] : 'N';
- if (c == '=' || seq_nt16_table[c] == seq_nt16_table[rb]) c = bam_is_rev(p->b)? ',' : '.';
- else c = bam_is_rev(p->b)? tolower(c) : toupper(c);
- } else {
- if (c == '=') c = bam_is_rev(p->b)? ',' : '.';
- else c = bam_is_rev(p->b)? tolower(c) : toupper(c);
+ int rb = pos < ref_len
+ ? seq_nt16_table[(uint8_t)(ref[pos])]
+ : 15/*N*/;
+ if (c == rb)
+ c = 0; // "=", which becomes . or ,
}
- putc(c, fp);
+ c = bam_is_rev(p->b)
+ ? seq_nt_str_lc[c]
+ : seq_nt_str_uc[c];
+ err |= kputc_(c, ks_seq) < 0;
+
if (m) {
int nm;
hts_base_mod mod[256];
if ((nm = bam_mods_at_qpos(p->b, p->qpos, m, mod, 256)) > 0) {
- putc('[', fp);
+ err |= kputc_('[', ks_seq) < 0;
int j;
for (j = 0; j < nm && j < 256; j++) {
char qual[20];
*qual = 0;
if (mod[j].modified_base < 0)
// ChEBI
- fprintf(fp, "%c(%d)%s", "+-"[mod[j].strand],
- -mod[j].modified_base, qual);
+ err |= ksprintf(ks_seq, "%c(%d)%s",
+ "+-"[mod[j].strand],
+ -mod[j].modified_base, qual) < 0;
else
- fprintf(fp, "%c%c%s", "+-"[mod[j].strand],
- mod[j].modified_base, qual);
+ err |= ksprintf(ks_seq, "%c%c%s", "+-"[mod[j].strand],
+ mod[j].modified_base, qual) < 0;
}
- putc(']', fp);
+ err |= kputc_(']', ks_seq) < 0;
}
}
- } else putc(p->is_refskip? (bam_is_rev(p->b)? '<' : '>') : ((bam_is_rev(p->b) && rev_del) ? '#' : '*'), fp);
+ } else {
+ err |= kputc_(p->is_refskip
+ ? (bam_is_rev(p->b)? '<' : '>')
+ : ((bam_is_rev(p->b) && rev_del) ? '#' : '*'),
+ ks_seq) < 0;
+ }
+
int del_len = -p->indel;
if (p->indel > 0) {
int len = bam_plp_insertion_mod(p, m && !no_ins_mods ? m : NULL,
- ks, &del_len);
+ ks_mod, &del_len);
if (len < 0) {
print_error("mpileup", "bam_plp_insertion() failed");
return -1;
}
if (no_ins < 2) {
- putc('+', fp);
- printw(len, fp);
+ err |= kputc_('+', ks_seq) < 0;
+ err |= kputuw(len, ks_seq) < 0;
}
if (!no_ins) {
+ kstring_t *ks = ks_mod;
if (bam_is_rev(p->b)) {
char pad = rev_del ? '#' : '*';
int in_mod = 0;
for (j = 0; j < ks->l; j++) {
if (ks->s[j] == '[') in_mod = 1;
else if (ks->s[j] == ']') in_mod = 0;
- putc(ks->s[j] != '*'
- ? (in_mod ? ks->s[j] : tolower(ks->s[j]))
- : pad, fp);
+ err |= kputc_(ks->s[j] != '*'
+ ? (in_mod ? ks->s[j] : tolower(ks->s[j]))
+ : pad, ks_seq) < 0;
}
} else {
int in_mod = 0;
for (j = 0; j < ks->l; j++) {
if (ks->s[j] == '[') in_mod = 1;
if (ks->s[j] == ']') in_mod = 0;
- putc(in_mod ? ks->s[j] : toupper(ks->s[j]), fp);
+ err |= kputc_(in_mod ? ks->s[j] : toupper(ks->s[j]),
+ ks_seq) < 0;
}
}
}
}
+
if (del_len > 0) {
if (no_del < 2)
- printw(-del_len, fp);
+ err |= kputw(-del_len, ks_seq) < 0;
if (!no_del) {
for (j = 1; j <= del_len; ++j) {
int c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N';
- putc(bam_is_rev(p->b)? tolower(c) : toupper(c), fp);
+ err |= kputc_(bam_is_rev(p->b)? tolower(c) : toupper(c),
+ ks_seq) < 0;
}
}
}
- if (!no_ends && p->is_tail) putc('$', fp);
- return 0;
+
+ if (!no_ends && p->is_tail)
+ err |= kputc_('$', ks_seq) < 0;
+
+ return -err;
}
#include "sample.h"
#define MPLP_PRINT_MAPQ_CHAR (1<<11)
#define MPLP_PRINT_QPOS (1<<12)
+// Start of struct active_cols elements
#define MPLP_PRINT_QNAME (1<<13)
#define MPLP_PRINT_FLAG (1<<14)
#define MPLP_PRINT_RNAME (1<<15)
#define MPLP_PRINT_TLEN (1<<21)
#define MPLP_PRINT_SEQ (1<<22)
#define MPLP_PRINT_QUAL (1<<23)
-#define MPLP_PRINT_MODS (1<<24)
-#define MPLP_PRINT_QPOS5 (1<<25)
+#define MPLP_PRINT_RLEN (1<<24)
+// Must occur after struct active_cols element list
+#define MPLP_PRINT_MODS (1<<25)
+#define MPLP_PRINT_QPOS5 (1<<26)
-#define MPLP_PRINT_LAST (1<<26) // terminator for loop
+#define MPLP_PRINT_LAST (1<<27) // terminator for loop
#define MPLP_MAX_DEPTH 8000
#define MPLP_MAX_INDEL_DEPTH 250
int supported;
};
- const struct active_cols colnames[11] = {
- {"QNAME", 1}, {"FLAG", 1}, {"RNAME", 1}, {"POS", 1}, {"MAPQ", 1}, {"CIGAR", 0}, {"RNEXT", 1}, {"PNEXT", 1}, {"TLEN", 0}, {"SEQ", 0}, {"QUAL", 0}
+ const struct active_cols colnames[12] = {
+ {"QNAME", 1}, {"FLAG", 1}, {"RNAME", 1}, {"POS", 1}, {"MAPQ", 1},
+ {"CIGAR", 0}, {"RNEXT", 1}, {"PNEXT", 1}, {"TLEN", 0}, {"SEQ", 0},
+ {"QUAL", 0}, {"RLEN", 1},
};
- int i, f = MPLP_PRINT_QNAME, colno = 11;
+ int i, f = MPLP_PRINT_QNAME, colno = sizeof(colnames)/sizeof(*colnames);
for (i = 0; i < colno; i++, f <<= 1)
if (colnames[i].supported)
khash_str2int_set(colhash, colnames[i].name, f);
* @param fn filenames
* @param fn_idx index filenames
*/
-static int mpileup(mplp_conf_t *conf, int n, char **fn, char **fn_idx)
+static int mpileup(mplp_conf_t *conf, int nfn, char **fn, char **fn_idx)
{
mplp_aux_t **data;
int i, tid, *n_plp, tid0 = 0, max_depth;
memset(&gplp, 0, sizeof(mplp_pileup_t));
memset(&buf, 0, sizeof(kstring_t));
- data = calloc(n, sizeof(mplp_aux_t*));
- plp = calloc(n, sizeof(bam_pileup1_t*));
- n_plp = calloc(n, sizeof(int));
+ data = calloc(nfn, sizeof(mplp_aux_t*));
+ plp = calloc(nfn, sizeof(bam_pileup1_t*));
+ n_plp = calloc(nfn, sizeof(int));
sm = bam_smpl_init();
- if (n == 0) {
+ if (nfn == 0) {
fprintf(samtools_stderr,"[%s] no input file/data given\n", __func__);
samtools_exit(EXIT_FAILURE);
}
// read the header of each file in the list and initialize data
refs_t *refs = NULL;
- for (i = 0; i < n; ++i) {
+ for (i = 0; i < nfn; ++i) {
sam_hdr_t *h_tmp;
data[i] = calloc(1, sizeof(mplp_aux_t));
data[i]->fp = sam_open_format(fn[i], "rb", &conf->ga.in);
data[i]->h = h;
}
}
- fprintf(samtools_stderr, "[%s] %d samples in %d input files\n", __func__, sm->n, n);
+ fprintf(samtools_stderr, "[%s] %d samples in %d input files\n",
+ __func__, sm->n, nfn);
pileup_fp = conf->output_fname? fopen(conf->output_fname, "w") : samtools_stdout;
}
// init pileup
- iter = bam_mplp_init(n, mplp_func, (void**)data);
+ iter = bam_mplp_init(nfn, mplp_func, (void**)data);
if (conf->flag & MPLP_PRINT_MODS) {
bam_mplp_constructor(iter, pileup_cd_create);
bam_mplp_destructor(iter, pileup_cd_destroy);
fprintf(samtools_stderr, "[%s] Max depth set to maximum value (%d)\n", __func__, INT_MAX);
} else {
max_depth = conf->max_depth;
- if ( max_depth * n > 1<<20 )
+ if ( max_depth * nfn > 1<<20 )
fprintf(samtools_stderr, "[%s] Combined max depth is above 1M. Potential memory hog!\n", __func__);
}
int ret;
int last_tid = -1;
hts_pos_t last_pos = -1;
+ int one_seq = 0;
// begin pileup
+ kstring_t ks_seq = KS_INITIALIZE;
+ kstring_t ks_mod = KS_INITIALIZE;
+ kstring_t ks_qual = KS_INITIALIZE;
while ( (ret=bam_mplp64_auto(iter, &tid, &pos, n_plp, plp)) > 0) {
+ one_seq = 1; // at least 1 output
if (conf->reg && (pos < beg0 || pos >= end0)) continue; // out of the region requested
- mplp_get_ref(data[0], tid, &ref, &ref_len);
- //printf("tid=%d len=%d ref=%p/%s\n", tid, ref_len, ref, ref);
if (conf->all) {
// Deal with missing portions of previous tids
while (tid > last_tid) {
while (++last_pos < sam_hdr_tid2len(h, last_tid)) {
if (conf->bed && bed_overlap(conf->bed, sam_hdr_tid2name(h, last_tid), last_pos, last_pos + 1) == 0)
continue;
- print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, n, ref, ref_len);
+ print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, nfn, ref, ref_len);
}
}
last_tid++;
last_pos = -1;
if (conf->all < 2)
break;
+ if (tid > last_tid)
+ // multiple missing references and -aa used
+ mplp_get_ref(data[0], last_tid, &ref, &ref_len);
}
}
+ mplp_get_ref(data[0], tid, &ref, &ref_len);
+
if (conf->all) {
// Deal with missing portion of current tid
while (++last_pos < pos) {
if (conf->reg && last_pos < beg0) continue; // out of range; skip
if (conf->bed && bed_overlap(conf->bed, sam_hdr_tid2name(h, tid), last_pos, last_pos + 1) == 0)
continue;
- print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, tid), last_pos, n, ref, ref_len);
+ print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, tid), last_pos, nfn, ref, ref_len);
}
last_tid = tid;
last_pos = pos;
if (conf->bed && tid >= 0 && !bed_overlap(conf->bed, sam_hdr_tid2name(h, tid), pos, pos+1)) continue;
fprintf(pileup_fp, "%s\t%"PRIhts_pos"\t%c", sam_hdr_tid2name(h, tid), pos + 1, (ref && pos < ref_len)? ref[pos] : 'N');
- for (i = 0; i < n; ++i) {
- int j, cnt;
+ for (i = 0; i < nfn; ++i) {
+ int j, cnt, err = 0;
+ ks_clear(&ks_seq);
+ ks_clear(&ks_qual);
+ ks_clear(&ks_mod);
for (j = cnt = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = plp[i] + j;
int c = p->qpos < p->b->core.l_qseq
? bam_get_qual(p->b)[p->qpos]
: 0;
- if (c >= conf->min_baseQ) ++cnt;
+ if (c >= conf->min_baseQ) {
+ // Build up seq
+ err |= pileup_seq(&ks_seq, plp[i] + j, pos, ref_len,
+ ref, &ks_mod, conf->rev_del,
+ conf->no_ins, conf->no_ins_mods,
+ conf->no_del, conf->no_ends) < 0;
+
+ // Build up qual
+ err |= kputc_(c+33 < 126 ? c+33 : 126, &ks_qual) < 0;
+ cnt++;
+ }
+ }
+ if (err) {
+ ret = 1;
+ goto fail;
}
fprintf(pileup_fp, "\t%d\t", cnt);
+
if (n_plp[i] == 0) {
fputs("*\t*", pileup_fp);
int flag_value = MPLP_PRINT_MAPQ_CHAR;
fputs("\t*", pileup_fp);
}
} else {
- int n = 0;
- kstring_t ks = KS_INITIALIZE;
- for (j = 0; j < n_plp[i]; ++j) {
- const bam_pileup1_t *p = plp[i] + j;
- int c = p->qpos < p->b->core.l_qseq
- ? bam_get_qual(p->b)[p->qpos]
- : 0;
- if (c >= conf->min_baseQ) {
- n++;
- if (pileup_seq(pileup_fp, plp[i] + j, pos, ref_len,
- ref, &ks, conf->rev_del,
- conf->no_ins, conf->no_ins_mods,
- conf->no_del, conf->no_ends) < 0) {
- ret = 1;
- goto fail;
- }
- }
+ if (ks_seq.l) {
+ fwrite(ks_seq.s, 1, ks_seq.l, pileup_fp);
+ } else {
+ putc('*', pileup_fp);
}
- if (!n) putc('*', pileup_fp);
-
- /* Print base qualities */
- n = 0;
- ks_free(&ks);
putc('\t', pileup_fp);
- for (j = 0; j < n_plp[i]; ++j) {
- const bam_pileup1_t *p = plp[i] + j;
- int c = p->qpos < p->b->core.l_qseq
- ? bam_get_qual(p->b)[p->qpos]
- : 0;
- if (c >= conf->min_baseQ) {
- c = c + 33 < 126? c + 33 : 126;
- putc(c, pileup_fp);
- n++;
- }
+
+ if (ks_qual.l) {
+ fwrite(ks_qual.s, 1, ks_qual.l, pileup_fp);
+ } else {
+ putc('*', pileup_fp);
}
- if (!n) putc('*', pileup_fp);
/* Print selected columns */
int flag_value = MPLP_PRINT_MAPQ_CHAR;
while(flag_value < MPLP_PRINT_LAST) {
if (flag_value != MPLP_PRINT_MODS
&& (conf->flag & flag_value)) {
- n = 0;
+ int n = 0;
putc('\t', pileup_fp);
for (j = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = &plp[i][j];
case MPLP_PRINT_PNEXT:
fprintf(pileup_fp, "%"PRId64, (int64_t) p->b->core.mpos + 1);
break;
+ case MPLP_PRINT_RLEN:
+ fprintf(pileup_fp, "%d", p->b->core.l_qseq);
+ break;
}
}
if (!n) putc('*', pileup_fp);
if (auxlist_p && auxlist_p->size) {
kliter_t(auxlist) *aux;
for (aux = kl_begin(auxlist_p); aux != kl_end(auxlist_p); aux = kl_next(aux)) {
- n = 0;
+ int n = 0; // NB shadows outer loop
putc('\t', pileup_fp);
for (j = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = &plp[i][j];
putc('\n', pileup_fp);
}
+ ks_free(&ks_seq);
+ ks_free(&ks_mod);
+ ks_free(&ks_qual);
+
if (ret < 0) {
print_error("mpileup", "error reading from input file");
ret = EXIT_FAILURE;
last_tid = tid0;
last_pos = beg0-1;
mplp_get_ref(data[0], tid0, &ref, &ref_len);
+ } else if (last_tid < 0 && !one_seq && conf->all > 1) {
+ last_tid = 0; // --aa on a blank file
}
- while (last_tid >= 0 && last_tid < sam_hdr_nref(h)) {
+ while (last_tid >= 0 && last_tid < sam_hdr_nref(h)) {
+ mplp_get_ref(data[0], last_tid, &ref, &ref_len);
while (++last_pos < sam_hdr_tid2len(h, last_tid)) {
if (last_pos >= end0) break;
if (conf->bed && bed_overlap(conf->bed, sam_hdr_tid2name(h, last_tid), last_pos, last_pos + 1) == 0)
continue;
- print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, n, ref, ref_len);
+ print_empty_pileup(pileup_fp, conf, sam_hdr_tid2name(h, last_tid), last_pos, nfn, ref, ref_len);
}
last_tid++;
last_pos = -1;
free(gplp.plp); free(gplp.n_plp); free(gplp.m_plp);
bam_mplp_destroy(iter);
sam_hdr_destroy(h);
- for (i = 0; i < n; ++i) {
+ for (i = 0; i < nfn; ++i) {
sam_close(data[i]->fp);
if (data[i]->iter) hts_itr_destroy(data[i]->iter);
free(data[i]);
/* bam_sort.c -- sorting and merging.
- Copyright (C) 2008-2023 Genome Research Ltd.
+ Copyright (C) 2008-2024 Genome Research Ltd.
Portions copyright (C) 2009-2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
typedef enum {Coordinate, QueryName, TagCoordinate, TagQueryName, MinHash, TemplateCoordinate} SamOrder;
static SamOrder g_sam_order = Coordinate;
+static int natural_sort = 1; // not ASCII, but alphanumeric: a12b > a7b
static char g_sort_tag[2] = {0,0};
#define is_digit(c) ((c)<='9' && (c)>='0')
static int strnum_cmp(const char *_a, const char *_b)
{
+ if (!natural_sort)
+ return strcmp(_a,_b);
+
const unsigned char *a = (const unsigned char*)_a, *b = (const unsigned char*)_b;
const unsigned char *pa = a, *pb = b;
while (*pa && *pb) {
case QueryName:
t = strnum_cmp(bam_get_qname(a.entry.bam_record), bam_get_qname(b.entry.bam_record));
if (t != 0) return t > 0;
- fa = a.entry.bam_record->core.flag & 0xc0;
- fb = b.entry.bam_record->core.flag & 0xc0;
+ fa = a.entry.bam_record->core.flag;
+ fb = b.entry.bam_record->core.flag;
+ // Sort order is READ1, READ2, (PRIMARY), SUPPLEMENTARY, SECONDARY
+ // Get the bits in this order so sort is a natural a-b
+ fa = ((fa&0xc0)<<8)|((fa&0x100)<<3)|((fa&0x800)>>3);
+ fb = ((fb&0xc0)<<8)|((fb&0x100)<<3)|((fb&0x800)>>3);
if (fa != fb) return fa > fb;
break;
case TagQueryName:
break;
}
- // This compares by position in the input file(s)
+ // This compares by position (i/idx'th read) in the input file(s)
if (a.i != b.i) return a.i > b.i;
return a.idx > b.idx;
}
* Create a merged_header_t struct.
*/
-static merged_header_t * init_merged_header() {
+static merged_header_t * init_merged_header(void) {
merged_header_t *merged_hdr;
merged_hdr = calloc(1, sizeof(*merged_hdr));
do {
dest->l = 0;
- ksprintf(dest, "%s-%0lX", prefix, lrand48());
+ ksprintf(dest, "%s-%08lX", prefix, lrand48());
iter = kh_get(cset, existing_ids, ks_str(dest));
} while (iter != kh_end(existing_ids));
// Make sure that there's enough memory for template coordinate keys, one per file to read
if (sam_order == TemplateCoordinate) {
if ((keys = malloc(sizeof(template_coordinate_keys_t))) == NULL) {
- print_error("sort", "could not allocate memory for the top-level keys");
+ print_error(cmd, "could not allocate memory for the top-level keys");
goto mem_fail;
}
keys->n = 0;
h->entry.u.tag = bam_aux_get(h->entry.bam_record, g_sort_tag);
} else if (g_sam_order == TemplateCoordinate) {
template_coordinate_key_t *key = template_coordinate_keys_get(keys, i); // get the next key to use
- h->entry.u.key = template_coordinate_key(heap->entry.bam_record, key, hout, lib_lookup); // update the key
- if (heap->entry.u.key == NULL) goto mem_fail; // key could not be created, error out
+ h->entry.u.key = template_coordinate_key(h->entry.bam_record, key, hout, lib_lookup); // update the key
+ if (h->entry.u.key == NULL) goto fail; // key could not be created, error out
} else {
h->entry.u.tag = NULL;
}
} else if (g_sam_order == TemplateCoordinate) {
template_coordinate_key_t *key = template_coordinate_keys_get(keys, heap->i); // get the next key to use
heap->entry.u.key = template_coordinate_key(heap->entry.bam_record, key, hout, lib_lookup); // update the key
- if (heap->entry.u.key == NULL) goto mem_fail; // key could not be created, error out
+ if (heap->entry.u.key == NULL) goto fail; // key could not be created, error out
} else {
heap->entry.u.tag = NULL;
}
bed_destroy(hreg);
free(RG); free(translation_tbl); free(fp); free(heap); free(iter); free(hdr);
if (sam_close(fpout) < 0) {
- print_error(cmd, "error closing output file");
+ print_error_errno(cmd, "error closing output file \"%s\"", out);
return -1;
}
+ if (keys != NULL) {
+ for (i = 0; i < keys->m; ++i) {
+ free(keys->buffers[i]);
+ }
+ free(keys->buffers);
+ free(keys);
+ }
+ lib_lookup_destroy(lib_lookup);
return 0;
mem_fail:
" or: samtools merge [options] <out.bam> <in1.bam> ... <inN.bam>\n"
"\n"
"Options:\n"
-" -n Input files are sorted by read name\n"
+" -n Input files are sorted by read name (natural)\n"
+" -N Input files are sorted by read name (ASCII)\n"
" -t TAG Input files are sorted by TAG value\n"
" -r Attach RG tag (inferred from file names)\n"
" -u Uncompressed BAM output\n"
return 0;
}
- while ((c = getopt_long(argc, argv, "h:nru1R:o:f@:l:cps:b:O:t:XL:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "h:nNru1R:o:f@:l:cps:b:O:t:XL:", lopts, NULL)) >= 0) {
switch (c) {
case 'r': flag |= MERGE_RG; break;
case 'f': flag |= MERGE_FORCE; break;
case 'h': fn_headers = optarg; break;
+ case 'N': natural_sort = 0; // fall through
case 'n': sam_order = QueryName; break;
case 'o': fnout = optarg; break;
case 't': sort_tag = optarg; break;
}
if (sam_close(fpout) < 0) {
- print_error(cmd, "error closing output file");
+ print_error_errno(cmd, "error closing output file \"%s\"", out);
return -1;
}
return 0;
if (g_sam_order == QueryName || g_sam_order == TagQueryName) {
int t = strnum_cmp(bam_get_qname(a.bam_record), bam_get_qname(b.bam_record));
if (t != 0) return t;
- return (int) (a.bam_record->core.flag&0xc0) - (int) (b.bam_record->core.flag&0xc0);
+ int af = a.bam_record->core.flag;
+ int bf = b.bam_record->core.flag;
+ // Sort order is READ1, READ2, (PRIMARY), SUPPLEMENTARY, SECONDARY
+ // Get the bits in this order so sort is a natural a-b
+ af = ((af&0xc0)<<8)|((af&0x100)<<3)|((af&0x800)>>3);
+ bf = ((bf&0xc0)<<8)|((bf&0x100)<<3)|((bf&0x800)>>3);
+ return af - bf;
} else {
pa = a.bam_record->core.tid;
pb = b.bam_record->core.tid;
break;
case MinHash:
worker_minhash(w);
- // no break, go to merge sort
+ // fall-through
default:
ks_mergesort(sort, w->buf_len, w->buf, 0);
}
break;
case QueryName:
new_so = "queryname";
+ new_ss = natural_sort
+ ? "queryname:natural"
+ : "queryname:lexicographical";
break;
case MinHash:
new_so = "coordinate";
" -I FILE Order minimisers by their position in FILE FASTA\n"
" -w INT Window size for minimiser indexing via -I ref.fa [100]\n"
" -H Squash homopolymers when computing minimiser\n"
-" -n Sort by read name (not compatible with samtools index command)\n"
+" -n Sort by read name (natural): cannot be used with samtools index\n"
+" -N Sort by read name (ASCII): cannot be used with samtools index\n"
" -t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set)\n"
" -o FILE Write final output to FILE rather than standard output\n"
" -T PREFIX Write temporary files to PREFIX.nnnn.bam\n"
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "l:m:no:O:T:@:t:MI:K:uRw:H", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "l:m:nNo:O:T:@:t:MI:K:uRw:H", lopts, NULL)) >= 0) {
switch (c) {
case 'o': fnout = optarg; o_seen = 1; break;
+ case 'N': natural_sort = 0; // fall through
case 'n': sam_order = QueryName; break;
case 't': by_tag = true; sort_tag = optarg; break;
case 'm': {
goto sort_end;
}
- if (ga.write_index && (sam_order == QueryName || sam_order == TagQueryName || sam_order == TagCoordinate || sam_order == TemplateCoordinate)) {
+ if (ga.write_index && sam_order != Coordinate) {
fprintf(stderr, "[W::bam_sort] Ignoring --write-index as it only works for position sorted files.\n");
ga.write_index = 0;
}
/* bam_sort.c -- sorting and merging.
- Copyright (C) 2008-2023 Genome Research Ltd.
+ Copyright (C) 2008-2024 Genome Research Ltd.
Portions copyright (C) 2009-2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
typedef enum {Coordinate, QueryName, TagCoordinate, TagQueryName, MinHash, TemplateCoordinate} SamOrder;
static SamOrder g_sam_order = Coordinate;
+static int natural_sort = 1; // not ASCII, but alphanumeric: a12b > a7b
static char g_sort_tag[2] = {0,0};
#define is_digit(c) ((c)<='9' && (c)>='0')
static int strnum_cmp(const char *_a, const char *_b)
{
+ if (!natural_sort)
+ return strcmp(_a,_b);
+
const unsigned char *a = (const unsigned char*)_a, *b = (const unsigned char*)_b;
const unsigned char *pa = a, *pb = b;
while (*pa && *pb) {
case QueryName:
t = strnum_cmp(bam_get_qname(a.entry.bam_record), bam_get_qname(b.entry.bam_record));
if (t != 0) return t > 0;
- fa = a.entry.bam_record->core.flag & 0xc0;
- fb = b.entry.bam_record->core.flag & 0xc0;
+ fa = a.entry.bam_record->core.flag;
+ fb = b.entry.bam_record->core.flag;
+ // Sort order is READ1, READ2, (PRIMARY), SUPPLEMENTARY, SECONDARY
+ // Get the bits in this order so sort is a natural a-b
+ fa = ((fa&0xc0)<<8)|((fa&0x100)<<3)|((fa&0x800)>>3);
+ fb = ((fb&0xc0)<<8)|((fb&0x100)<<3)|((fb&0x800)>>3);
if (fa != fb) return fa > fb;
break;
case TagQueryName:
break;
}
- // This compares by position in the input file(s)
+ // This compares by position (i/idx'th read) in the input file(s)
if (a.i != b.i) return a.i > b.i;
return a.idx > b.idx;
}
* Create a merged_header_t struct.
*/
-static merged_header_t * init_merged_header() {
+static merged_header_t * init_merged_header(void) {
merged_header_t *merged_hdr;
merged_hdr = calloc(1, sizeof(*merged_hdr));
do {
dest->l = 0;
- ksprintf(dest, "%s-%0lX", prefix, lrand48());
+ ksprintf(dest, "%s-%08lX", prefix, lrand48());
iter = kh_get(cset, existing_ids, ks_str(dest));
} while (iter != kh_end(existing_ids));
// Make sure that there's enough memory for template coordinate keys, one per file to read
if (sam_order == TemplateCoordinate) {
if ((keys = malloc(sizeof(template_coordinate_keys_t))) == NULL) {
- print_error("sort", "could not allocate memory for the top-level keys");
+ print_error(cmd, "could not allocate memory for the top-level keys");
goto mem_fail;
}
keys->n = 0;
h->entry.u.tag = bam_aux_get(h->entry.bam_record, g_sort_tag);
} else if (g_sam_order == TemplateCoordinate) {
template_coordinate_key_t *key = template_coordinate_keys_get(keys, i); // get the next key to use
- h->entry.u.key = template_coordinate_key(heap->entry.bam_record, key, hout, lib_lookup); // update the key
- if (heap->entry.u.key == NULL) goto mem_fail; // key could not be created, error out
+ h->entry.u.key = template_coordinate_key(h->entry.bam_record, key, hout, lib_lookup); // update the key
+ if (h->entry.u.key == NULL) goto fail; // key could not be created, error out
} else {
h->entry.u.tag = NULL;
}
} else if (g_sam_order == TemplateCoordinate) {
template_coordinate_key_t *key = template_coordinate_keys_get(keys, heap->i); // get the next key to use
heap->entry.u.key = template_coordinate_key(heap->entry.bam_record, key, hout, lib_lookup); // update the key
- if (heap->entry.u.key == NULL) goto mem_fail; // key could not be created, error out
+ if (heap->entry.u.key == NULL) goto fail; // key could not be created, error out
} else {
heap->entry.u.tag = NULL;
}
bed_destroy(hreg);
free(RG); free(translation_tbl); free(fp); free(heap); free(iter); free(hdr);
if (sam_close(fpout) < 0) {
- print_error(cmd, "error closing output file");
+ print_error_errno(cmd, "error closing output file \"%s\"", out);
return -1;
}
+ if (keys != NULL) {
+ for (i = 0; i < keys->m; ++i) {
+ free(keys->buffers[i]);
+ }
+ free(keys->buffers);
+ free(keys);
+ }
+ lib_lookup_destroy(lib_lookup);
return 0;
mem_fail:
" or: samtools merge [options] <out.bam> <in1.bam> ... <inN.bam>\n"
"\n"
"Options:\n"
-" -n Input files are sorted by read name\n"
+" -n Input files are sorted by read name (natural)\n"
+" -N Input files are sorted by read name (ASCII)\n"
" -t TAG Input files are sorted by TAG value\n"
" -r Attach RG tag (inferred from file names)\n"
" -u Uncompressed BAM output\n"
return 0;
}
- while ((c = getopt_long(argc, argv, "h:nru1R:o:f@:l:cps:b:O:t:XL:", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "h:nNru1R:o:f@:l:cps:b:O:t:XL:", lopts, NULL)) >= 0) {
switch (c) {
case 'r': flag |= MERGE_RG; break;
case 'f': flag |= MERGE_FORCE; break;
case 'h': fn_headers = optarg; break;
+ case 'N': natural_sort = 0; // fall through
case 'n': sam_order = QueryName; break;
case 'o': fnout = optarg; break;
case 't': sort_tag = optarg; break;
}
if (sam_close(fpout) < 0) {
- print_error(cmd, "error closing output file");
+ print_error_errno(cmd, "error closing output file \"%s\"", out);
return -1;
}
return 0;
if (g_sam_order == QueryName || g_sam_order == TagQueryName) {
int t = strnum_cmp(bam_get_qname(a.bam_record), bam_get_qname(b.bam_record));
if (t != 0) return t;
- return (int) (a.bam_record->core.flag&0xc0) - (int) (b.bam_record->core.flag&0xc0);
+ int af = a.bam_record->core.flag;
+ int bf = b.bam_record->core.flag;
+ // Sort order is READ1, READ2, (PRIMARY), SUPPLEMENTARY, SECONDARY
+ // Get the bits in this order so sort is a natural a-b
+ af = ((af&0xc0)<<8)|((af&0x100)<<3)|((af&0x800)>>3);
+ bf = ((bf&0xc0)<<8)|((bf&0x100)<<3)|((bf&0x800)>>3);
+ return af - bf;
} else {
pa = a.bam_record->core.tid;
pb = b.bam_record->core.tid;
break;
case MinHash:
worker_minhash(w);
- // no break, go to merge sort
+ // fall-through
default:
ks_mergesort(sort, w->buf_len, w->buf, 0);
}
break;
case QueryName:
new_so = "queryname";
+ new_ss = natural_sort
+ ? "queryname:natural"
+ : "queryname:lexicographical";
break;
case MinHash:
new_so = "coordinate";
" -I FILE Order minimisers by their position in FILE FASTA\n"
" -w INT Window size for minimiser indexing via -I ref.fa [100]\n"
" -H Squash homopolymers when computing minimiser\n"
-" -n Sort by read name (not compatible with samtools index command)\n"
+" -n Sort by read name (natural): cannot be used with samtools index\n"
+" -N Sort by read name (ASCII): cannot be used with samtools index\n"
" -t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set)\n"
" -o FILE Write final output to FILE rather than standard output\n"
" -T PREFIX Write temporary files to PREFIX.nnnn.bam\n"
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "l:m:no:O:T:@:t:MI:K:uRw:H", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "l:m:nNo:O:T:@:t:MI:K:uRw:H", lopts, NULL)) >= 0) {
switch (c) {
case 'o': fnout = optarg; o_seen = 1; break;
+ case 'N': natural_sort = 0; // fall through
case 'n': sam_order = QueryName; break;
case 't': by_tag = true; sort_tag = optarg; break;
case 'm': {
goto sort_end;
}
- if (ga.write_index && (sam_order == QueryName || sam_order == TagQueryName || sam_order == TagCoordinate || sam_order == TemplateCoordinate)) {
+ if (ga.write_index && sam_order != Coordinate) {
fprintf(samtools_stderr, "[W::bam_sort] Ignoring --write-index as it only works for position sorted files.\n");
ga.write_index = 0;
}
/* bam_split.c -- split subcommand.
- Copyright (C) 2013-2016,2018-2019,2023 Genome Research Ltd.
+ Copyright (C) 2013-2016,2018-2019,2023,2024 Genome Research Ltd.
Author: Martin Pollard <mp15@sanger.ac.uk>
const char *unaccounted_header_name;
const char *unaccounted_name;
const char *output_format_string;
+ const char *tag;
+ long max_split;
bool verbose;
int no_pg;
+ int zero_pad;
sam_global_args ga;
};
+#define DEFAULT_MAX_SPLIT 100
+
typedef struct parsed_opts parsed_opts_t;
struct state {
sam_hdr_t* merged_input_header;
samFile* unaccounted_file;
sam_hdr_t* unaccounted_header;
+ char *unaccounted_idx_fn;
size_t output_count;
- char** rg_id;
- char **rg_index_file_name;
- char **rg_output_file_name;
- samFile** rg_output_file;
- sam_hdr_t** rg_output_header;
- kh_c2i_t* rg_hash;
+ char **tag_vals;
+ char **index_file_name;
+ char **output_file_name;
+ samFile **output_file;
+ sam_hdr_t **output_header;
+ kh_c2i_t* tag_val_hash;
htsThreadPool p;
int write_index;
};
"Usage: samtools split [-u <unaccounted.bam>] [-h <unaccounted_header.sam>]\n"
" [-f <format_string>] [-v] <merged.bam>\n"
"Options:\n"
-" -f STRING output filename format string [\"%%*_%%#.%%.\"]\n"
-" -u FILE1 put reads with no RG tag or an unrecognised RG tag in FILE1\n"
-" -h FILE2 ... and override the header with FILE2 (-u file only)\n"
-" -v verbose output\n"
-" --no-PG do not add a PG line\n");
+" -f STRING output filename format string [\"%%*_%%#.%%.\"]\n"
+" -u FILE1 put left-over reads in FILE1\n"
+" -h FILE2 ... and override the header with FILE2 (-u file only)\n"
+" -d TAG split by TAG value. TAG value must be a string.\n"
+" -p NUMBER zero-pad numbers in filenames to NUMBER digits\n"
+" -M,--max-split NUM limit number of output files from -d to NUM [%d]\n"
+" -v verbose output\n"
+" --no-PG do not add a PG line\n",
+ DEFAULT_MAX_SPLIT);
sam_global_opt_help(write_to, "-....@..");
fprintf(write_to,
"\n"
"Format string expansions:\n"
" %%%% %%\n"
" %%* basename\n"
-" %%# @RG index\n"
-" %%! @RG ID\n"
+" %%# index (of @RG in the header, or count of TAG values seen so far)\n"
+" %%! @RG ID or TAG value\n"
" %%. filename extension for output format\n"
);
}
{
if (argc == 1) { usage(stdout); return NULL; }
- const char *optstring = "vf:h:u:@:";
+ const char *optstring = "vf:h:u:d:M:p:@:";
+ char *default_format_string = "%*_%#.%.";
static const struct option lopts[] = {
SAM_OPT_GLOBAL_OPTIONS('-', 0, 0, 0, 0, '@'),
{"no-PG", no_argument, NULL, 1},
+ {"max-split", required_argument, NULL, 'M'},
+ {"zero-pad", required_argument, NULL, 'p'},
{ NULL, 0, NULL, 0 }
};
parsed_opts_t* retval = calloc(sizeof(parsed_opts_t), 1);
if (! retval ) { perror("cannot allocate option parsing memory"); return NULL; }
+ retval->max_split = DEFAULT_MAX_SPLIT;
sam_global_args_init(&retval->ga);
int opt;
case 'u':
retval->unaccounted_name = optarg;
break;
+ case 'd':
+ retval->tag = optarg;
+ default_format_string = "%*_%!.%.";
+ break;
+ case 'M': {
+ char *end = optarg;
+ retval->max_split = strtol(optarg, &end, 10);
+ if (*optarg == '\0' || *end != '\0' || retval->max_split == 0) {
+ print_error("split", "Invalid -M argument: \"%s\"", optarg);
+ free(retval);
+ return NULL;
+ }
+ if (retval->max_split < 0) // No limit requested
+ retval->max_split = LONG_MAX;
+ break;
+ }
+ case 'p': {
+ char *end = optarg;
+ unsigned long val = strtoul(optarg, &end, 10);
+ if (*optarg == '\0' || *end != '\0' || val > 20) {
+ print_error("split", "Invalid -p argument: \"%s\"", optarg);
+ free(retval);
+ return NULL;
+ }
+ retval->zero_pad = (int) val;
+ break;
+ }
case 1:
retval->no_pg = 1;
break;
+
default:
if (parse_sam_global_opt(opt, optarg, lopts, &retval->ga) == 0) break;
/* else fall-through */
}
}
- if (retval->output_format_string == NULL) retval->output_format_string = "%*_%#.%.";
+ if (retval->output_format_string == NULL) retval->output_format_string = default_format_string;
argc -= optind;
argv += optind;
}
// Expands a output filename format string
-static char* expand_format_string(const char* format_string, const char* basename, const char* rg_id, const int rg_idx, const htsFormat *format)
+static char* expand_format_string(const char* format_string,
+ const char* basename, const char* tag_val,
+ const int file_idx, const int zero_pad,
+ const htsFormat *format)
{
kstring_t str = { 0, 0, NULL };
const char* pointer = format_string;
if (kputs(basename, &str) < 0) goto memfail;
break;
case '#':
- if (kputl(rg_idx, &str) < 0) goto memfail;
+ if (zero_pad == 0) {
+ if (kputl(file_idx, &str) < 0) goto memfail;
+ } else {
+ if (ksprintf(&str, "%0*d", zero_pad, file_idx) < 0)
+ goto memfail;
+ }
break;
case '!':
- if (kputs(rg_id, &str) < 0) goto memfail;
+ if (kputs(tag_val, &str) < 0) goto memfail;
break;
case '.':
// Only really need to cope with sam, bam, cram
return 0;
}
+static int grow_output_lists(state_t *state, size_t count) {
+ char **new_list = realloc(state->tag_vals, count * sizeof(char *));
+ if (!new_list)
+ return -1;
+ state->tag_vals = new_list;
+ new_list = realloc(state->index_file_name, count * sizeof(char *));
+ if (!new_list)
+ return -1;
+ state->index_file_name = new_list;
+ new_list = realloc(state->output_file_name, count * sizeof(char *));
+ if (!new_list)
+ return -1;
+ state->output_file_name = new_list;
+ samFile **new_file = realloc(state->output_file,
+ count * sizeof(samFile *));
+ if (!new_file)
+ return -1;
+ state->output_file = new_file;
+ sam_hdr_t **new_hdr = realloc(state->output_header,
+ count * sizeof(sam_hdr_t *));
+ if (!new_hdr)
+ return -1;
+ state->output_header = new_hdr;
+ return 0;
+}
+
+static khiter_t prep_sam_file(parsed_opts_t *opts, state_t *state,
+ const char *tag, const char *arg_list,
+ int is_rg) {
+ char *input_base_name = NULL, *new_file_name = NULL, *tag_val = NULL;
+ char *new_idx_fn = NULL;
+ sam_hdr_t *new_hdr = NULL;
+ samFile *new_sam_file = NULL;
+
+ khiter_t i = kh_get_c2i(state->tag_val_hash, tag);
+ if (i != kh_end(state->tag_val_hash)) {
+ return i;
+ }
+ // create new file
+ if (grow_output_lists(state, state->output_count + 1) != 0) {
+ print_error_errno("split", "Couldn't grow output lists");
+ return kh_end(state->tag_val_hash);
+ }
+ tag_val = strdup(tag);
+ if (!tag_val) {
+ print_error_errno("split", "Couldn't copy tag value");
+ return kh_end(state->tag_val_hash);
+ }
+ char *dirsep = strrchr(opts->merged_input_name, '/');
+ input_base_name = strdup(dirsep? dirsep+1 : opts->merged_input_name);
+ if (!input_base_name) {
+ print_error_errno("split", "Filename parsing failed");
+ goto fail;
+ }
+
+ char* extension = strrchr(input_base_name, '.');
+ if (extension) *extension = '\0';
+
+ new_file_name = expand_format_string(opts->output_format_string,
+ input_base_name, tag,
+ kh_size(state->tag_val_hash),
+ opts->zero_pad, &opts->ga.out);
+ if (!new_file_name) {
+ print_error_errno("split", "Filename creation failed");
+ goto fail;
+ }
+
+ new_hdr = sam_hdr_dup(state->merged_input_header);
+ if (!new_hdr) {
+ print_error_errno("split", "Duplicating header for file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+ if (!opts->no_pg && sam_hdr_add_pg(new_hdr, "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL)) {
+ print_error_errno("split", "Adding PG line to file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+
+ if (is_rg) {
+ // If here, we've found an RG:Z: tag without a corresponding @RG
+ // line in the header.
+ if (sam_hdr_remove_lines(new_hdr, "RG", "ID", NULL) != 0) {
+ print_error_errno("split",
+ "Failed to remove @RG lines from file \"%s\"",
+ new_file_name);
+ goto fail;
+ }
+ if (sam_hdr_add_line(new_hdr, "RG", "ID", tag_val, NULL) != 0) {
+ print_error_errno("split",
+ "Failed to add @RG line to file \"%s\"",
+ new_file_name);
+ goto fail;
+ }
+ }
+
+ char outmode[4] = "w";
+ sam_open_mode(outmode + 1, new_file_name, NULL);
+ new_sam_file = sam_open_format(new_file_name, outmode, &opts->ga.out);
+ if (!new_sam_file) {
+ print_error_errno("split", "Opening filename for writing \"%s\" failed", new_file_name);
+ goto fail;
+ }
+ if (state->p.pool)
+ hts_set_opt(new_sam_file, HTS_OPT_THREAD_POOL, &state->p);
+
+ if (sam_hdr_write(new_sam_file, new_hdr) != 0) {
+ print_error_errno("split", "Couldn't write header to \"%s\"",
+ new_file_name);
+ goto fail;
+ }
+
+ if (state->write_index) {
+ new_idx_fn = auto_index(new_sam_file, new_file_name, new_hdr);
+ if (!new_idx_fn) {
+ print_error_errno("split", "Creating index file for file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+ }
+
+ int ret = -1;
+ i = kh_put_c2i(state->tag_val_hash, tag_val, &ret);
+ if (ret < 0) {
+ print_error_errno("split", "Adding file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+
+ kh_val(state->tag_val_hash, i) = state->output_count;
+ state->tag_vals[state->output_count] = tag_val;
+ state->index_file_name[state->output_count] = new_idx_fn;
+ state->output_file_name[state->output_count] = new_file_name;
+ state->output_file[state->output_count] = new_sam_file;
+ state->output_header[state->output_count] = new_hdr;
+ state->output_count++;
+ free(input_base_name);
+ return i;
+
+ fail:
+ free(input_base_name);
+ free(new_file_name);
+ free(tag_val);
+ free(new_idx_fn);
+ if (new_hdr)
+ sam_hdr_destroy(new_hdr);
+ if (new_sam_file)
+ sam_close(new_sam_file);
+ return kh_end(state->tag_val_hash);
+}
+
// Set the initial state
static state_t* init(parsed_opts_t* opts, const char *arg_list)
{
cleanup_state(retval, false);
return NULL;
}
+ retval->write_index = opts->ga.write_index;
if (opts->unaccounted_name) {
if (opts->unaccounted_header_name) {
} else {
retval->unaccounted_header = sam_hdr_dup(retval->merged_input_header);
if (!opts->no_pg && sam_hdr_add_pg(retval->unaccounted_header, "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL)) {
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL)) {
print_error("split", "Could not rewrite header for \"%s\"", opts->unaccounted_name);
cleanup_state(retval, false);
return NULL;
hts_set_opt(retval->unaccounted_file, HTS_OPT_THREAD_POOL, &retval->p);
}
- // Open output files for RGs
- if (!count_RG(retval->merged_input_header, &retval->output_count, &retval->rg_id)) return NULL;
- if (opts->verbose) fprintf(stderr, "@RG's found %zu\n",retval->output_count);
+ int is_rg = !opts->tag || strcmp(opts->tag, "RG") == 0;
+ if (is_rg) {
+ if (!count_RG(retval->merged_input_header,
+ &retval->output_count, &retval->tag_vals)) {
+ cleanup_state(retval, false);
+ return NULL;
+ }
+ if (opts->verbose)
+ fprintf(stderr, "@RG's found %zu\n",retval->output_count);
+ } else {
+ retval->output_count = 0;
+ }
+
// Prevent calloc(0, size);
size_t num = retval->output_count ? retval->output_count : 1;
- retval->rg_index_file_name = (char **)calloc(num, sizeof(char *));
- retval->rg_output_file_name = (char **)calloc(num, sizeof(char *));
- retval->rg_output_file = (samFile**)calloc(num, sizeof(samFile*));
- retval->rg_output_header = (sam_hdr_t**)calloc(num, sizeof(sam_hdr_t*));
- retval->rg_hash = kh_init_c2i();
- if (!retval->rg_output_file_name || !retval->rg_output_file || !retval->rg_output_header ||
- !retval->rg_hash || !retval->rg_index_file_name) {
+ retval->index_file_name = (char **)calloc(num, sizeof(char *));
+ retval->output_file_name = (char **)calloc(num, sizeof(char *));
+ retval->output_file = (samFile**)calloc(num, sizeof(samFile*));
+ retval->output_header = (sam_hdr_t**)calloc(num, sizeof(sam_hdr_t*));
+ retval->tag_val_hash = kh_init_c2i();
+ if (!retval->output_file_name || !retval->output_file || !retval->output_header ||
+ !retval->tag_val_hash || !retval->index_file_name) {
print_error_errno("split", "Could not initialise output file array");
cleanup_state(retval, false);
return NULL;
}
+ if (!is_rg)
+ return retval; // Done for this case - outputs will be opened later
+ // Adjust max_split if too small for the read-groups listed in the header
+ if (opts->max_split < retval->output_count)
+ opts->max_split = retval->output_count;
+
+ // Open output files for RGs
char* dirsep = strrchr(opts->merged_input_name, '/');
char* input_base_name = strdup(dirsep? dirsep+1 : opts->merged_input_name);
if (!input_base_name) {
output_filename = expand_format_string(opts->output_format_string,
input_base_name,
- retval->rg_id[i], i,
- &opts->ga.out);
+ retval->tag_vals[i], i,
+ opts->zero_pad, &opts->ga.out);
if ( output_filename == NULL ) {
cleanup_state(retval, false);
return NULL;
}
- retval->rg_output_file_name[i] = output_filename;
-
+ retval->output_file_name[i] = output_filename;
sam_open_mode(outmode + 1, output_filename, NULL);
- retval->rg_output_file[i] = sam_open_format(output_filename, outmode, &opts->ga.out);
+ retval->output_file[i] = sam_open_format(output_filename, outmode, &opts->ga.out);
- if (retval->rg_output_file[i] == NULL) {
+ if (retval->output_file[i] == NULL) {
print_error_errno("split", "Could not open \"%s\"", output_filename);
cleanup_state(retval, false);
free(input_base_name);
return NULL;
}
if (retval->p.pool)
- hts_set_opt(retval->rg_output_file[i], HTS_OPT_THREAD_POOL, &retval->p);
+ hts_set_opt(retval->output_file[i], HTS_OPT_THREAD_POOL, &retval->p);
// Record index in hash
int ret;
- khiter_t iter = kh_put_c2i(retval->rg_hash, retval->rg_id[i], &ret);
+ khiter_t iter = kh_put_c2i(retval->tag_val_hash, retval->tag_vals[i], &ret);
if (ret < 0) {
print_error_errno("split", "Couldn't add @RG ID to look-up table");
cleanup_state(retval, false);
free(input_base_name);
return NULL;
}
- kh_val(retval->rg_hash,iter) = i;
+ kh_val(retval->tag_val_hash,iter) = i;
// Set and edit header
- retval->rg_output_header[i] = sam_hdr_dup(retval->merged_input_header);
- if (sam_hdr_remove_except(retval->rg_output_header[i], "RG", "ID", retval->rg_id[i]) ||
- (!opts->no_pg &&
- sam_hdr_add_pg(retval->rg_output_header[i], "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL))) {
+ retval->output_header[i] = sam_hdr_dup(retval->merged_input_header);
+ if (sam_hdr_remove_except(retval->output_header[i], "RG", "ID", retval->tag_vals[i]) ||
+ (!opts->no_pg &&
+ sam_hdr_add_pg(retval->output_header[i], "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL))) {
print_error("split", "Could not rewrite header for \"%s\"", output_filename);
cleanup_state(retval, false);
free(input_base_name);
}
free(input_base_name);
- retval->write_index = opts->ga.write_index;
return retval;
}
-static bool split(state_t* state)
+static bool split(state_t* state, parsed_opts_t *opts, char *arg_list)
{
- if (state->unaccounted_file && sam_hdr_write(state->unaccounted_file, state->unaccounted_header) != 0) {
- print_error_errno("split", "Could not write output file header");
- return false;
- }
- size_t i;
- for (i = 0; i < state->output_count; i++) {
- if (sam_hdr_write(state->rg_output_file[i], state->rg_output_header[i]) != 0) {
- print_error_errno("split", "Could not write file header to \"%s\"", state->rg_output_file_name[i]);
+ int is_rg = !opts->tag || strcmp(opts->tag, "RG") == 0;
+ if (state->unaccounted_file) {
+ if (sam_hdr_write(state->unaccounted_file, state->unaccounted_header) != 0) {
+ print_error_errno("split", "Could not write output file header");
return false;
}
- if (state->write_index) {
- state->rg_index_file_name[i] = auto_index(state->rg_output_file[i],
- state->rg_output_file_name[i],
- state->rg_output_header[i]);
- if (!state->rg_index_file_name[i]) {
- print_error_errno("split", "Could not create index for file \"%s\"", state->rg_output_file_name[i]);
+ if (opts->ga.write_index) {
+ state->unaccounted_idx_fn = auto_index(state->unaccounted_file,
+ opts->unaccounted_name,
+ state->unaccounted_header);
+ if (!state->unaccounted_idx_fn) {
+ print_error_errno("split",
+ "Creating index file for file \"%s\" failed",
+ opts->unaccounted_name);
return false;
}
}
}
-
bam1_t* file_read = bam_init1();
// Read the first record
int r;
}
}
+ if (is_rg) {
+ size_t i;
+ for (i = 0; i < state->output_count; i++) {
+ if (sam_hdr_write(state->output_file[i], state->output_header[i]) != 0) {
+ print_error_errno("split", "Could not write file header to \"%s\"", state->output_file_name[i]);
+ goto error;
+ }
+ if (state->write_index) {
+ state->index_file_name[i] = auto_index(state->output_file[i],
+ state->output_file_name[i],
+ state->output_header[i]);
+ if (!state->index_file_name[i]) {
+ print_error_errno("split", "Could not create index for file \"%s\"", state->output_file_name[i]);
+ goto error;
+ }
+ }
+ }
+ }
while (file_read != NULL) {
// Get RG tag from read and look it up in hash to find file to output it to
- uint8_t* tag = bam_aux_get(file_read, "RG");
+ uint8_t* tag = bam_aux_get(file_read, is_rg ? "RG" : opts->tag);
+ char *val = NULL;
+ char number[28];
khiter_t iter;
- if ( tag != NULL ) {
- char* rg = bam_aux2Z(tag);
- iter = kh_get_c2i(state->rg_hash, rg);
+ if (tag) {
+ switch (*tag) {
+ case 'Z': case 'H':
+ val = bam_aux2Z(tag);
+ break;
+ case 'c': case 'C': case 's': case 'S': case 'i': case 'I':
+ if (opts->zero_pad == 0) {
+ snprintf(number, sizeof(number), "%"PRId64, bam_aux2i(tag));
+ } else {
+ int64_t v = bam_aux2i(tag);
+ snprintf(number, sizeof(number), "%0*"PRId64,
+ v < 0 ? opts->zero_pad + 1 : opts->zero_pad, v);
+ }
+ val = number;
+ break;
+ default:
+ break;
+ }
+ }
+ if ( val != NULL ) {
+ iter = kh_get_c2i(state->tag_val_hash, val);
} else {
- iter = kh_end(state->rg_hash);
+ iter = kh_end(state->tag_val_hash);
+ }
+
+ // Check for opts->tag here instead of !is_rg so we open new
+ // files if the user specified '-d RG' and we find a RG:Z: value
+ // that wasn't listed in the header. If the '-d' option is
+ // not used, we don't open a file to preserve existing behaviour.
+ if (opts->tag && val && iter == kh_end(state->tag_val_hash)
+ && state->output_count < opts->max_split) {
+ // Need to open a new output file
+ iter = prep_sam_file(opts, state, val, arg_list, is_rg);
+ if (iter == kh_end(state->tag_val_hash)) { // Open failed
+ print_error("split",
+ "Could not create output file for tag \"%s:%s\"",
+ opts->tag, bam_aux2Z(tag));
+ goto error;
+
+ }
}
// Write the read out to correct file
- if (iter != kh_end(state->rg_hash)) {
+ if (iter != kh_end(state->tag_val_hash)) {
// if found write to the appropriate untangled bam
- int i = kh_val(state->rg_hash,iter);
- if (sam_write1(state->rg_output_file[i], state->rg_output_header[i], file_read) < 0) {
- print_error_errno("split", "Could not write to \"%s\"", state->rg_output_file_name[i]);
- bam_destroy1(file_read);
- return false;
+ int i = kh_val(state->tag_val_hash,iter);
+ if (sam_write1(state->output_file[i], state->output_header[i], file_read) < 0) {
+ print_error_errno("split", "Could not write to \"%s\"", state->output_file_name[i]);
+ goto error;
}
} else {
// otherwise write to the unaccounted bam if there is one or fail
if (tag) {
fprintf(stderr, "Read \"%s\" with unaccounted for tag \"%s\".\n", bam_get_qname(file_read), bam_aux2Z(tag));
} else {
- fprintf(stderr, "Read \"%s\" has no RG tag.\n", bam_get_qname(file_read));
+ fprintf(stderr, "Read \"%s\" has no %s tag.\n",
+ bam_get_qname(file_read), is_rg ? "RG" : opts->tag);
}
- bam_destroy1(file_read);
- return false;
+ goto error;
} else {
if (sam_write1(state->unaccounted_file, state->unaccounted_header, file_read) < 0) {
print_error_errno("split", "Could not write to unaccounted output file");
- bam_destroy1(file_read);
- return false;
+ goto error;
}
}
}
}
if (state->write_index) {
+ size_t i;
for (i = 0; i < state->output_count; i++) {
- if (sam_idx_save(state->rg_output_file[i]) < 0) {
- print_error_errno("split", "writing index failed");
+ if (sam_idx_save(state->output_file[i]) < 0) {
+ print_error_errno("split", "writing index \"%s\" failed",
+ state->index_file_name[i]);
+ return false;
+ }
+ }
+ if (state->unaccounted_file) {
+ if (sam_idx_save(state->unaccounted_file) < 0) {
+ print_error_errno("split", "writing index \"%s\" failed",
+ state->unaccounted_idx_fn);
return false;
}
- free(state->rg_index_file_name[i]);
}
}
return true;
+error:
+ bam_destroy1(file_read);
+ return false;
}
static int cleanup_state(state_t* status, bool check_close)
sam_close(status->merged_input_file);
size_t i;
for (i = 0; i < status->output_count; i++) {
- if (status->rg_output_header && status->rg_output_header[i])
- sam_hdr_destroy(status->rg_output_header[i]);
- if (status->rg_output_file && status->rg_output_file[i]) {
- if (sam_close(status->rg_output_file[i]) < 0 && check_close) {
- print_error("split", "Error on closing output file \"%s\"", status->rg_output_file_name[i]);
+ if (status->output_header && status->output_header[i])
+ sam_hdr_destroy(status->output_header[i]);
+ if (status->output_file && status->output_file[i]) {
+ if (sam_close(status->output_file[i]) < 0 && check_close) {
+ print_error("split", "Error on closing output file \"%s\"", status->output_file_name[i]);
ret = -1;
}
}
- if (status->rg_id) free(status->rg_id[i]);
- if (status->rg_output_file_name) free(status->rg_output_file_name[i]);
+ if (status->tag_vals) free(status->tag_vals[i]);
+ if (status->output_file_name) free(status->output_file_name[i]);
+ if (status->index_file_name[i]) free(status->index_file_name[i]);
}
if (status->merged_input_header)
sam_hdr_destroy(status->merged_input_header);
- free(status->rg_output_header);
- free(status->rg_output_file);
- free(status->rg_output_file_name);
- free(status->rg_index_file_name);
- kh_destroy_c2i(status->rg_hash);
- free(status->rg_id);
+ free(status->output_header);
+ free(status->output_file);
+ free(status->output_file_name);
+ free(status->index_file_name);
+ free(status->unaccounted_idx_fn);
+ kh_destroy_c2i(status->tag_val_hash);
+
+ free(status->tag_vals);
if (status->p.pool)
hts_tpool_destroy(status->p.pool);
free(status);
state_t* status = init(opts, arg_list);
if (!status) goto cleanup_opts;
- if (!split(status)) {
+ if (!split(status, opts, arg_list)) {
cleanup_state(status, false);
goto cleanup_opts;
}
/* bam_split.c -- split subcommand.
- Copyright (C) 2013-2016,2018-2019,2023 Genome Research Ltd.
+ Copyright (C) 2013-2016,2018-2019,2023,2024 Genome Research Ltd.
Author: Martin Pollard <mp15@sanger.ac.uk>
const char *unaccounted_header_name;
const char *unaccounted_name;
const char *output_format_string;
+ const char *tag;
+ long max_split;
bool verbose;
int no_pg;
+ int zero_pad;
sam_global_args ga;
};
+#define DEFAULT_MAX_SPLIT 100
+
typedef struct parsed_opts parsed_opts_t;
struct state {
sam_hdr_t* merged_input_header;
samFile* unaccounted_file;
sam_hdr_t* unaccounted_header;
+ char *unaccounted_idx_fn;
size_t output_count;
- char** rg_id;
- char **rg_index_file_name;
- char **rg_output_file_name;
- samFile** rg_output_file;
- sam_hdr_t** rg_output_header;
- kh_c2i_t* rg_hash;
+ char **tag_vals;
+ char **index_file_name;
+ char **output_file_name;
+ samFile **output_file;
+ sam_hdr_t **output_header;
+ kh_c2i_t* tag_val_hash;
htsThreadPool p;
int write_index;
};
"Usage: samtools split [-u <unaccounted.bam>] [-h <unaccounted_header.sam>]\n"
" [-f <format_string>] [-v] <merged.bam>\n"
"Options:\n"
-" -f STRING output filename format string [\"%%*_%%#.%%.\"]\n"
-" -u FILE1 put reads with no RG tag or an unrecognised RG tag in FILE1\n"
-" -h FILE2 ... and override the header with FILE2 (-u file only)\n"
-" -v verbose output\n"
-" --no-PG do not add a PG line\n");
+" -f STRING output filename format string [\"%%*_%%#.%%.\"]\n"
+" -u FILE1 put left-over reads in FILE1\n"
+" -h FILE2 ... and override the header with FILE2 (-u file only)\n"
+" -d TAG split by TAG value. TAG value must be a string.\n"
+" -p NUMBER zero-pad numbers in filenames to NUMBER digits\n"
+" -M,--max-split NUM limit number of output files from -d to NUM [%d]\n"
+" -v verbose output\n"
+" --no-PG do not add a PG line\n",
+ DEFAULT_MAX_SPLIT);
sam_global_opt_help(write_to, "-....@..");
fprintf(write_to,
"\n"
"Format string expansions:\n"
" %%%% %%\n"
" %%* basename\n"
-" %%# @RG index\n"
-" %%! @RG ID\n"
+" %%# index (of @RG in the header, or count of TAG values seen so far)\n"
+" %%! @RG ID or TAG value\n"
" %%. filename extension for output format\n"
);
}
{
if (argc == 1) { usage(samtools_stdout); return NULL; }
- const char *optstring = "vf:h:u:@:";
+ const char *optstring = "vf:h:u:d:M:p:@:";
+ char *default_format_string = "%*_%#.%.";
static const struct option lopts[] = {
SAM_OPT_GLOBAL_OPTIONS('-', 0, 0, 0, 0, '@'),
{"no-PG", no_argument, NULL, 1},
+ {"max-split", required_argument, NULL, 'M'},
+ {"zero-pad", required_argument, NULL, 'p'},
{ NULL, 0, NULL, 0 }
};
parsed_opts_t* retval = calloc(sizeof(parsed_opts_t), 1);
if (! retval ) { perror("cannot allocate option parsing memory"); return NULL; }
+ retval->max_split = DEFAULT_MAX_SPLIT;
sam_global_args_init(&retval->ga);
int opt;
case 'u':
retval->unaccounted_name = optarg;
break;
+ case 'd':
+ retval->tag = optarg;
+ default_format_string = "%*_%!.%.";
+ break;
+ case 'M': {
+ char *end = optarg;
+ retval->max_split = strtol(optarg, &end, 10);
+ if (*optarg == '\0' || *end != '\0' || retval->max_split == 0) {
+ print_error("split", "Invalid -M argument: \"%s\"", optarg);
+ free(retval);
+ return NULL;
+ }
+ if (retval->max_split < 0) // No limit requested
+ retval->max_split = LONG_MAX;
+ break;
+ }
+ case 'p': {
+ char *end = optarg;
+ unsigned long val = strtoul(optarg, &end, 10);
+ if (*optarg == '\0' || *end != '\0' || val > 20) {
+ print_error("split", "Invalid -p argument: \"%s\"", optarg);
+ free(retval);
+ return NULL;
+ }
+ retval->zero_pad = (int) val;
+ break;
+ }
case 1:
retval->no_pg = 1;
break;
+
default:
if (parse_sam_global_opt(opt, optarg, lopts, &retval->ga) == 0) break;
/* else fall-through */
}
}
- if (retval->output_format_string == NULL) retval->output_format_string = "%*_%#.%.";
+ if (retval->output_format_string == NULL) retval->output_format_string = default_format_string;
argc -= optind;
argv += optind;
}
// Expands a output filename format string
-static char* expand_format_string(const char* format_string, const char* basename, const char* rg_id, const int rg_idx, const htsFormat *format)
+static char* expand_format_string(const char* format_string,
+ const char* basename, const char* tag_val,
+ const int file_idx, const int zero_pad,
+ const htsFormat *format)
{
kstring_t str = { 0, 0, NULL };
const char* pointer = format_string;
if (kputs(basename, &str) < 0) goto memfail;
break;
case '#':
- if (kputl(rg_idx, &str) < 0) goto memfail;
+ if (zero_pad == 0) {
+ if (kputl(file_idx, &str) < 0) goto memfail;
+ } else {
+ if (ksprintf(&str, "%0*d", zero_pad, file_idx) < 0)
+ goto memfail;
+ }
break;
case '!':
- if (kputs(rg_id, &str) < 0) goto memfail;
+ if (kputs(tag_val, &str) < 0) goto memfail;
break;
case '.':
// Only really need to cope with sam, bam, cram
return 0;
}
+static int grow_output_lists(state_t *state, size_t count) {
+ char **new_list = realloc(state->tag_vals, count * sizeof(char *));
+ if (!new_list)
+ return -1;
+ state->tag_vals = new_list;
+ new_list = realloc(state->index_file_name, count * sizeof(char *));
+ if (!new_list)
+ return -1;
+ state->index_file_name = new_list;
+ new_list = realloc(state->output_file_name, count * sizeof(char *));
+ if (!new_list)
+ return -1;
+ state->output_file_name = new_list;
+ samFile **new_file = realloc(state->output_file,
+ count * sizeof(samFile *));
+ if (!new_file)
+ return -1;
+ state->output_file = new_file;
+ sam_hdr_t **new_hdr = realloc(state->output_header,
+ count * sizeof(sam_hdr_t *));
+ if (!new_hdr)
+ return -1;
+ state->output_header = new_hdr;
+ return 0;
+}
+
+static khiter_t prep_sam_file(parsed_opts_t *opts, state_t *state,
+ const char *tag, const char *arg_list,
+ int is_rg) {
+ char *input_base_name = NULL, *new_file_name = NULL, *tag_val = NULL;
+ char *new_idx_fn = NULL;
+ sam_hdr_t *new_hdr = NULL;
+ samFile *new_sam_file = NULL;
+
+ khiter_t i = kh_get_c2i(state->tag_val_hash, tag);
+ if (i != kh_end(state->tag_val_hash)) {
+ return i;
+ }
+ // create new file
+ if (grow_output_lists(state, state->output_count + 1) != 0) {
+ print_error_errno("split", "Couldn't grow output lists");
+ return kh_end(state->tag_val_hash);
+ }
+ tag_val = strdup(tag);
+ if (!tag_val) {
+ print_error_errno("split", "Couldn't copy tag value");
+ return kh_end(state->tag_val_hash);
+ }
+ char *dirsep = strrchr(opts->merged_input_name, '/');
+ input_base_name = strdup(dirsep? dirsep+1 : opts->merged_input_name);
+ if (!input_base_name) {
+ print_error_errno("split", "Filename parsing failed");
+ goto fail;
+ }
+
+ char* extension = strrchr(input_base_name, '.');
+ if (extension) *extension = '\0';
+
+ new_file_name = expand_format_string(opts->output_format_string,
+ input_base_name, tag,
+ kh_size(state->tag_val_hash),
+ opts->zero_pad, &opts->ga.out);
+ if (!new_file_name) {
+ print_error_errno("split", "Filename creation failed");
+ goto fail;
+ }
+
+ new_hdr = sam_hdr_dup(state->merged_input_header);
+ if (!new_hdr) {
+ print_error_errno("split", "Duplicating header for file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+ if (!opts->no_pg && sam_hdr_add_pg(new_hdr, "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL)) {
+ print_error_errno("split", "Adding PG line to file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+
+ if (is_rg) {
+ // If here, we've found an RG:Z: tag without a corresponding @RG
+ // line in the header.
+ if (sam_hdr_remove_lines(new_hdr, "RG", "ID", NULL) != 0) {
+ print_error_errno("split",
+ "Failed to remove @RG lines from file \"%s\"",
+ new_file_name);
+ goto fail;
+ }
+ if (sam_hdr_add_line(new_hdr, "RG", "ID", tag_val, NULL) != 0) {
+ print_error_errno("split",
+ "Failed to add @RG line to file \"%s\"",
+ new_file_name);
+ goto fail;
+ }
+ }
+
+ char outmode[4] = "w";
+ sam_open_mode(outmode + 1, new_file_name, NULL);
+ new_sam_file = sam_open_format(new_file_name, outmode, &opts->ga.out);
+ if (!new_sam_file) {
+ print_error_errno("split", "Opening filename for writing \"%s\" failed", new_file_name);
+ goto fail;
+ }
+ if (state->p.pool)
+ hts_set_opt(new_sam_file, HTS_OPT_THREAD_POOL, &state->p);
+
+ if (sam_hdr_write(new_sam_file, new_hdr) != 0) {
+ print_error_errno("split", "Couldn't write header to \"%s\"",
+ new_file_name);
+ goto fail;
+ }
+
+ if (state->write_index) {
+ new_idx_fn = auto_index(new_sam_file, new_file_name, new_hdr);
+ if (!new_idx_fn) {
+ print_error_errno("split", "Creating index file for file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+ }
+
+ int ret = -1;
+ i = kh_put_c2i(state->tag_val_hash, tag_val, &ret);
+ if (ret < 0) {
+ print_error_errno("split", "Adding file \"%s\" failed", new_file_name);
+ goto fail;
+ }
+
+ kh_val(state->tag_val_hash, i) = state->output_count;
+ state->tag_vals[state->output_count] = tag_val;
+ state->index_file_name[state->output_count] = new_idx_fn;
+ state->output_file_name[state->output_count] = new_file_name;
+ state->output_file[state->output_count] = new_sam_file;
+ state->output_header[state->output_count] = new_hdr;
+ state->output_count++;
+ free(input_base_name);
+ return i;
+
+ fail:
+ free(input_base_name);
+ free(new_file_name);
+ free(tag_val);
+ free(new_idx_fn);
+ if (new_hdr)
+ sam_hdr_destroy(new_hdr);
+ if (new_sam_file)
+ sam_close(new_sam_file);
+ return kh_end(state->tag_val_hash);
+}
+
// Set the initial state
static state_t* init(parsed_opts_t* opts, const char *arg_list)
{
cleanup_state(retval, false);
return NULL;
}
+ retval->write_index = opts->ga.write_index;
if (opts->unaccounted_name) {
if (opts->unaccounted_header_name) {
} else {
retval->unaccounted_header = sam_hdr_dup(retval->merged_input_header);
if (!opts->no_pg && sam_hdr_add_pg(retval->unaccounted_header, "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL)) {
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL)) {
print_error("split", "Could not rewrite header for \"%s\"", opts->unaccounted_name);
cleanup_state(retval, false);
return NULL;
hts_set_opt(retval->unaccounted_file, HTS_OPT_THREAD_POOL, &retval->p);
}
- // Open output files for RGs
- if (!count_RG(retval->merged_input_header, &retval->output_count, &retval->rg_id)) return NULL;
- if (opts->verbose) fprintf(samtools_stderr, "@RG's found %zu\n",retval->output_count);
+ int is_rg = !opts->tag || strcmp(opts->tag, "RG") == 0;
+ if (is_rg) {
+ if (!count_RG(retval->merged_input_header,
+ &retval->output_count, &retval->tag_vals)) {
+ cleanup_state(retval, false);
+ return NULL;
+ }
+ if (opts->verbose)
+ fprintf(samtools_stderr, "@RG's found %zu\n",retval->output_count);
+ } else {
+ retval->output_count = 0;
+ }
+
// Prevent calloc(0, size);
size_t num = retval->output_count ? retval->output_count : 1;
- retval->rg_index_file_name = (char **)calloc(num, sizeof(char *));
- retval->rg_output_file_name = (char **)calloc(num, sizeof(char *));
- retval->rg_output_file = (samFile**)calloc(num, sizeof(samFile*));
- retval->rg_output_header = (sam_hdr_t**)calloc(num, sizeof(sam_hdr_t*));
- retval->rg_hash = kh_init_c2i();
- if (!retval->rg_output_file_name || !retval->rg_output_file || !retval->rg_output_header ||
- !retval->rg_hash || !retval->rg_index_file_name) {
+ retval->index_file_name = (char **)calloc(num, sizeof(char *));
+ retval->output_file_name = (char **)calloc(num, sizeof(char *));
+ retval->output_file = (samFile**)calloc(num, sizeof(samFile*));
+ retval->output_header = (sam_hdr_t**)calloc(num, sizeof(sam_hdr_t*));
+ retval->tag_val_hash = kh_init_c2i();
+ if (!retval->output_file_name || !retval->output_file || !retval->output_header ||
+ !retval->tag_val_hash || !retval->index_file_name) {
print_error_errno("split", "Could not initialise output file array");
cleanup_state(retval, false);
return NULL;
}
+ if (!is_rg)
+ return retval; // Done for this case - outputs will be opened later
+ // Adjust max_split if too small for the read-groups listed in the header
+ if (opts->max_split < retval->output_count)
+ opts->max_split = retval->output_count;
+
+ // Open output files for RGs
char* dirsep = strrchr(opts->merged_input_name, '/');
char* input_base_name = strdup(dirsep? dirsep+1 : opts->merged_input_name);
if (!input_base_name) {
output_filename = expand_format_string(opts->output_format_string,
input_base_name,
- retval->rg_id[i], i,
- &opts->ga.out);
+ retval->tag_vals[i], i,
+ opts->zero_pad, &opts->ga.out);
if ( output_filename == NULL ) {
cleanup_state(retval, false);
return NULL;
}
- retval->rg_output_file_name[i] = output_filename;
-
+ retval->output_file_name[i] = output_filename;
sam_open_mode(outmode + 1, output_filename, NULL);
- retval->rg_output_file[i] = sam_open_format(output_filename, outmode, &opts->ga.out);
+ retval->output_file[i] = sam_open_format(output_filename, outmode, &opts->ga.out);
- if (retval->rg_output_file[i] == NULL) {
+ if (retval->output_file[i] == NULL) {
print_error_errno("split", "Could not open \"%s\"", output_filename);
cleanup_state(retval, false);
free(input_base_name);
return NULL;
}
if (retval->p.pool)
- hts_set_opt(retval->rg_output_file[i], HTS_OPT_THREAD_POOL, &retval->p);
+ hts_set_opt(retval->output_file[i], HTS_OPT_THREAD_POOL, &retval->p);
// Record index in hash
int ret;
- khiter_t iter = kh_put_c2i(retval->rg_hash, retval->rg_id[i], &ret);
+ khiter_t iter = kh_put_c2i(retval->tag_val_hash, retval->tag_vals[i], &ret);
if (ret < 0) {
print_error_errno("split", "Couldn't add @RG ID to look-up table");
cleanup_state(retval, false);
free(input_base_name);
return NULL;
}
- kh_val(retval->rg_hash,iter) = i;
+ kh_val(retval->tag_val_hash,iter) = i;
// Set and edit header
- retval->rg_output_header[i] = sam_hdr_dup(retval->merged_input_header);
- if (sam_hdr_remove_except(retval->rg_output_header[i], "RG", "ID", retval->rg_id[i]) ||
- (!opts->no_pg &&
- sam_hdr_add_pg(retval->rg_output_header[i], "samtools",
- "VN", samtools_version(),
- arg_list ? "CL": NULL,
- arg_list ? arg_list : NULL,
- NULL))) {
+ retval->output_header[i] = sam_hdr_dup(retval->merged_input_header);
+ if (sam_hdr_remove_except(retval->output_header[i], "RG", "ID", retval->tag_vals[i]) ||
+ (!opts->no_pg &&
+ sam_hdr_add_pg(retval->output_header[i], "samtools",
+ "VN", samtools_version(),
+ arg_list ? "CL": NULL,
+ arg_list ? arg_list : NULL,
+ NULL))) {
print_error("split", "Could not rewrite header for \"%s\"", output_filename);
cleanup_state(retval, false);
free(input_base_name);
}
free(input_base_name);
- retval->write_index = opts->ga.write_index;
return retval;
}
-static bool split(state_t* state)
+static bool split(state_t* state, parsed_opts_t *opts, char *arg_list)
{
- if (state->unaccounted_file && sam_hdr_write(state->unaccounted_file, state->unaccounted_header) != 0) {
- print_error_errno("split", "Could not write output file header");
- return false;
- }
- size_t i;
- for (i = 0; i < state->output_count; i++) {
- if (sam_hdr_write(state->rg_output_file[i], state->rg_output_header[i]) != 0) {
- print_error_errno("split", "Could not write file header to \"%s\"", state->rg_output_file_name[i]);
+ int is_rg = !opts->tag || strcmp(opts->tag, "RG") == 0;
+ if (state->unaccounted_file) {
+ if (sam_hdr_write(state->unaccounted_file, state->unaccounted_header) != 0) {
+ print_error_errno("split", "Could not write output file header");
return false;
}
- if (state->write_index) {
- state->rg_index_file_name[i] = auto_index(state->rg_output_file[i],
- state->rg_output_file_name[i],
- state->rg_output_header[i]);
- if (!state->rg_index_file_name[i]) {
- print_error_errno("split", "Could not create index for file \"%s\"", state->rg_output_file_name[i]);
+ if (opts->ga.write_index) {
+ state->unaccounted_idx_fn = auto_index(state->unaccounted_file,
+ opts->unaccounted_name,
+ state->unaccounted_header);
+ if (!state->unaccounted_idx_fn) {
+ print_error_errno("split",
+ "Creating index file for file \"%s\" failed",
+ opts->unaccounted_name);
return false;
}
}
}
-
bam1_t* file_read = bam_init1();
// Read the first record
int r;
}
}
+ if (is_rg) {
+ size_t i;
+ for (i = 0; i < state->output_count; i++) {
+ if (sam_hdr_write(state->output_file[i], state->output_header[i]) != 0) {
+ print_error_errno("split", "Could not write file header to \"%s\"", state->output_file_name[i]);
+ goto error;
+ }
+ if (state->write_index) {
+ state->index_file_name[i] = auto_index(state->output_file[i],
+ state->output_file_name[i],
+ state->output_header[i]);
+ if (!state->index_file_name[i]) {
+ print_error_errno("split", "Could not create index for file \"%s\"", state->output_file_name[i]);
+ goto error;
+ }
+ }
+ }
+ }
while (file_read != NULL) {
// Get RG tag from read and look it up in hash to find file to output it to
- uint8_t* tag = bam_aux_get(file_read, "RG");
+ uint8_t* tag = bam_aux_get(file_read, is_rg ? "RG" : opts->tag);
+ char *val = NULL;
+ char number[28];
khiter_t iter;
- if ( tag != NULL ) {
- char* rg = bam_aux2Z(tag);
- iter = kh_get_c2i(state->rg_hash, rg);
+ if (tag) {
+ switch (*tag) {
+ case 'Z': case 'H':
+ val = bam_aux2Z(tag);
+ break;
+ case 'c': case 'C': case 's': case 'S': case 'i': case 'I':
+ if (opts->zero_pad == 0) {
+ snprintf(number, sizeof(number), "%"PRId64, bam_aux2i(tag));
+ } else {
+ int64_t v = bam_aux2i(tag);
+ snprintf(number, sizeof(number), "%0*"PRId64,
+ v < 0 ? opts->zero_pad + 1 : opts->zero_pad, v);
+ }
+ val = number;
+ break;
+ default:
+ break;
+ }
+ }
+ if ( val != NULL ) {
+ iter = kh_get_c2i(state->tag_val_hash, val);
} else {
- iter = kh_end(state->rg_hash);
+ iter = kh_end(state->tag_val_hash);
+ }
+
+ // Check for opts->tag here instead of !is_rg so we open new
+ // files if the user specified '-d RG' and we find a RG:Z: value
+ // that wasn't listed in the header. If the '-d' option is
+ // not used, we don't open a file to preserve existing behaviour.
+ if (opts->tag && val && iter == kh_end(state->tag_val_hash)
+ && state->output_count < opts->max_split) {
+ // Need to open a new output file
+ iter = prep_sam_file(opts, state, val, arg_list, is_rg);
+ if (iter == kh_end(state->tag_val_hash)) { // Open failed
+ print_error("split",
+ "Could not create output file for tag \"%s:%s\"",
+ opts->tag, bam_aux2Z(tag));
+ goto error;
+
+ }
}
// Write the read out to correct file
- if (iter != kh_end(state->rg_hash)) {
+ if (iter != kh_end(state->tag_val_hash)) {
// if found write to the appropriate untangled bam
- int i = kh_val(state->rg_hash,iter);
- if (sam_write1(state->rg_output_file[i], state->rg_output_header[i], file_read) < 0) {
- print_error_errno("split", "Could not write to \"%s\"", state->rg_output_file_name[i]);
- bam_destroy1(file_read);
- return false;
+ int i = kh_val(state->tag_val_hash,iter);
+ if (sam_write1(state->output_file[i], state->output_header[i], file_read) < 0) {
+ print_error_errno("split", "Could not write to \"%s\"", state->output_file_name[i]);
+ goto error;
}
} else {
// otherwise write to the unaccounted bam if there is one or fail
if (tag) {
fprintf(samtools_stderr, "Read \"%s\" with unaccounted for tag \"%s\".\n", bam_get_qname(file_read), bam_aux2Z(tag));
} else {
- fprintf(samtools_stderr, "Read \"%s\" has no RG tag.\n", bam_get_qname(file_read));
+ fprintf(samtools_stderr, "Read \"%s\" has no %s tag.\n",
+ bam_get_qname(file_read), is_rg ? "RG" : opts->tag);
}
- bam_destroy1(file_read);
- return false;
+ goto error;
} else {
if (sam_write1(state->unaccounted_file, state->unaccounted_header, file_read) < 0) {
print_error_errno("split", "Could not write to unaccounted output file");
- bam_destroy1(file_read);
- return false;
+ goto error;
}
}
}
}
if (state->write_index) {
+ size_t i;
for (i = 0; i < state->output_count; i++) {
- if (sam_idx_save(state->rg_output_file[i]) < 0) {
- print_error_errno("split", "writing index failed");
+ if (sam_idx_save(state->output_file[i]) < 0) {
+ print_error_errno("split", "writing index \"%s\" failed",
+ state->index_file_name[i]);
+ return false;
+ }
+ }
+ if (state->unaccounted_file) {
+ if (sam_idx_save(state->unaccounted_file) < 0) {
+ print_error_errno("split", "writing index \"%s\" failed",
+ state->unaccounted_idx_fn);
return false;
}
- free(state->rg_index_file_name[i]);
}
}
return true;
+error:
+ bam_destroy1(file_read);
+ return false;
}
static int cleanup_state(state_t* status, bool check_close)
sam_close(status->merged_input_file);
size_t i;
for (i = 0; i < status->output_count; i++) {
- if (status->rg_output_header && status->rg_output_header[i])
- sam_hdr_destroy(status->rg_output_header[i]);
- if (status->rg_output_file && status->rg_output_file[i]) {
- if (sam_close(status->rg_output_file[i]) < 0 && check_close) {
- print_error("split", "Error on closing output file \"%s\"", status->rg_output_file_name[i]);
+ if (status->output_header && status->output_header[i])
+ sam_hdr_destroy(status->output_header[i]);
+ if (status->output_file && status->output_file[i]) {
+ if (sam_close(status->output_file[i]) < 0 && check_close) {
+ print_error("split", "Error on closing output file \"%s\"", status->output_file_name[i]);
ret = -1;
}
}
- if (status->rg_id) free(status->rg_id[i]);
- if (status->rg_output_file_name) free(status->rg_output_file_name[i]);
+ if (status->tag_vals) free(status->tag_vals[i]);
+ if (status->output_file_name) free(status->output_file_name[i]);
+ if (status->index_file_name[i]) free(status->index_file_name[i]);
}
if (status->merged_input_header)
sam_hdr_destroy(status->merged_input_header);
- free(status->rg_output_header);
- free(status->rg_output_file);
- free(status->rg_output_file_name);
- free(status->rg_index_file_name);
- kh_destroy_c2i(status->rg_hash);
- free(status->rg_id);
+ free(status->output_header);
+ free(status->output_file);
+ free(status->output_file_name);
+ free(status->index_file_name);
+ free(status->unaccounted_idx_fn);
+ kh_destroy_c2i(status->tag_val_hash);
+
+ free(status->tag_vals);
if (status->p.pool)
hts_tpool_destroy(status->p.pool);
free(status);
state_t* status = init(opts, arg_list);
if (!status) goto cleanup_opts;
- if (!split(status)) {
+ if (!split(status, opts, arg_list)) {
cleanup_state(status, false);
goto cleanup_opts;
}
int main_reset(int argc, char *argv[]);
int main_cram_size(int argc, char *argv[]);
-const char *samtools_version()
+const char *samtools_version(void)
{
return SAMTOOLS_VERSION;
}
static void long_version(void) {
printf("samtools %s\n"
"Using htslib %s\n"
- "Copyright (C) 2023 Genome Research Ltd.\n",
+ "Copyright (C) 2024 Genome Research Ltd.\n",
samtools_version(), hts_version());
printf("\nSamtools compilation details:\n");
fprintf(stderr, "[main] unrecognized command '%s'\n", argv[1]);
return 1;
}
+
+ // For subcommands that may have produced substantial output on stdout,
+ // make a final check for delayed I/O errors. Ignore EBADF as other code
+ // may have already closed stdout.
+ if (fclose(stdout) != 0 && errno != EBADF) {
+ print_error_errno(argv[1], "closing standard output failed");
+ return 1;
+ }
+
return ret;
}
int main_reset(int argc, char *argv[]);
int main_cram_size(int argc, char *argv[]);
-const char *samtools_version()
+const char *samtools_version(void)
{
return SAMTOOLS_VERSION;
}
static void long_version(void) {
fprintf(samtools_stdout, "samtools %s\n"
"Using htslib %s\n"
- "Copyright (C) 2023 Genome Research Ltd.\n",
+ "Copyright (C) 2024 Genome Research Ltd.\n",
samtools_version(), hts_version());
fprintf(samtools_stdout, "\nSamtools compilation details:\n");
fprintf(samtools_stderr, "[main] unrecognized command '%s'\n", argv[1]);
return 1;
}
+
+ // For subcommands that may have produced substantial output on samtools_stdout,
+ // make a final check for delayed I/O errors. Ignore EBADF as other code
+ // may have already closed samtools_stdout.
+ if (0) { //if (fclose(samtools_stdout) != 0 && errno != EBADF) {
+ print_error_errno(argv[1], "closing standard output failed");
+ return 1;
+ }
+
return ret;
}
/* bedcov.c -- bedcov subcommand.
Copyright (C) 2012 Broad Institute.
- Copyright (C) 2013-2014, 2018-2022 Genome Research Ltd.
+ Copyright (C) 2013-2014, 2018-2022, 2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk>
#include "htslib/kseq.h"
KSTREAM_INIT(gzFile, gzread, 16384)
-#define DEFAULT_DEPTH 64000
+#define DEFAULT_DEPTH INT_MAX
typedef struct {
htsFile *fp;
const bam_pileup1_t **plp;
int usage = 0, has_index_file = 0;
uint32_t flags = (BAM_FUNMAP | BAM_FSECONDARY | BAM_FQCFAIL | BAM_FDUP);
- int tflags = 0, min_depth = -1;
+ int tflags = 0, min_depth = -1, max_depth = DEFAULT_DEPTH, print_header=0;
sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
static const struct option lopts[] = {
{"min-MQ", required_argument, NULL, 'Q'},
{"min-mq", required_argument, NULL, 'Q'},
+ {"max-depth", required_argument, NULL, 'd'+1000},
SAM_OPT_GLOBAL_OPTIONS('-', 0, '-', '-', 0, '-'),
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "Q:Xg:G:jd:c", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "Q:Xg:G:jd:Hc", lopts, NULL)) >= 0) {
switch (c) {
case 'Q': min_mapQ = atoi(optarg); break;
case 'X': has_index_file = 1; break;
case 'c': do_rcount = 1; break;
+ case 'H': print_header = 1; break;
case 'g':
tflags = bam_str2flag(optarg);
if (tflags < 0 || tflags > ((BAM_FSUPPLEMENTARY << 1) - 1)) {
break;
case 'j': skip_DN = 1; break;
case 'd': min_depth = atoi(optarg); break;
+ case 'd'+1000: max_depth = atoi(optarg); break;
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
/* else fall-through */
case '?': usage = 1; break;
fprintf(stderr, " -G <flags> add the specified flags to the set used to filter out reads\n"
" The default set is UNMAP,SECONDARY,QCFAIL,DUP or 0x704\n");
fprintf(stderr, " -j do not include deletions (D) and ref skips (N) in bedcov computation\n");
- fprintf(stderr, " -d <int> depth threshold. Number of reference bases with coverage above and"
+ fprintf(stderr, " --max-depth <int> sets the maximum depth used in the mpileup algorithm\n");
+ fprintf(stderr, " -d <int> depth threshold. Number of reference bases with coverage above and\n"
" including this value will be displayed in a separate column\n");
fprintf(stderr, " -c add an additional column showing read count\n");
+ fprintf(stderr, " -H print a comment/header line with column information.\n");
sam_global_opt_help(stderr, "-.--.--.");
return 1;
}
print_error_errno("bedcov", "can't open BED file '%s'", argv[optind]);
return 2;
}
+
+ if (print_header) {
+ printf("#chrom\tstart\tend");
+ for (i = 0; i < n; ++i) {
+ printf("\t%s_cov", argv[i+optind+1]);
+ }
+ if (min_depth >= 0) {
+ for (i = 0; i < n; ++i)
+ printf("\t%s_depth", argv[i+optind+1]);
+ }
+ if (do_rcount) {
+ for (i = 0; i < n; ++i)
+ printf("\t%s_count", argv[i+optind+1]);
+ }
+ putchar('\n');
+ }
+
ks = ks_init(fp);
n_plp = calloc(n, sizeof(int));
plp = calloc(n, sizeof(bam_pileup1_t*));
}
mplp = bam_mplp_init(n, read_bam, (void**)aux);
- if (min_depth > DEFAULT_DEPTH)
+ if (min_depth > max_depth)
+ // NB: never happens given current DEFAULT_DEPTH of INT_MAX
bam_mplp_set_maxcnt(mplp, min_depth);
else
- bam_mplp_set_maxcnt(mplp, DEFAULT_DEPTH);
+ bam_mplp_set_maxcnt(mplp, max_depth);
memset(cnt, 0, sizeof(*cnt) * n);
if (min_depth >= 0) memset(pcov, 0, sizeof(*pcov) * n);
/* bedcov.c -- bedcov subcommand.
Copyright (C) 2012 Broad Institute.
- Copyright (C) 2013-2014, 2018-2022 Genome Research Ltd.
+ Copyright (C) 2013-2014, 2018-2022, 2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk>
#include "htslib/kseq.h"
KSTREAM_INIT(gzFile, gzread, 16384)
-#define DEFAULT_DEPTH 64000
+#define DEFAULT_DEPTH INT_MAX
typedef struct {
htsFile *fp;
const bam_pileup1_t **plp;
int usage = 0, has_index_file = 0;
uint32_t flags = (BAM_FUNMAP | BAM_FSECONDARY | BAM_FQCFAIL | BAM_FDUP);
- int tflags = 0, min_depth = -1;
+ int tflags = 0, min_depth = -1, max_depth = DEFAULT_DEPTH, print_header=0;
sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
static const struct option lopts[] = {
{"min-MQ", required_argument, NULL, 'Q'},
{"min-mq", required_argument, NULL, 'Q'},
+ {"max-depth", required_argument, NULL, 'd'+1000},
SAM_OPT_GLOBAL_OPTIONS('-', 0, '-', '-', 0, '-'),
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "Q:Xg:G:jd:c", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "Q:Xg:G:jd:Hc", lopts, NULL)) >= 0) {
switch (c) {
case 'Q': min_mapQ = atoi(optarg); break;
case 'X': has_index_file = 1; break;
case 'c': do_rcount = 1; break;
+ case 'H': print_header = 1; break;
case 'g':
tflags = bam_str2flag(optarg);
if (tflags < 0 || tflags > ((BAM_FSUPPLEMENTARY << 1) - 1)) {
break;
case 'j': skip_DN = 1; break;
case 'd': min_depth = atoi(optarg); break;
+ case 'd'+1000: max_depth = atoi(optarg); break;
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
/* else fall-through */
case '?': usage = 1; break;
fprintf(samtools_stderr, " -G <flags> add the specified flags to the set used to filter out reads\n"
" The default set is UNMAP,SECONDARY,QCFAIL,DUP or 0x704\n");
fprintf(samtools_stderr, " -j do not include deletions (D) and ref skips (N) in bedcov computation\n");
- fprintf(samtools_stderr, " -d <int> depth threshold. Number of reference bases with coverage above and"
+ fprintf(samtools_stderr, " --max-depth <int> sets the maximum depth used in the mpileup algorithm\n");
+ fprintf(samtools_stderr, " -d <int> depth threshold. Number of reference bases with coverage above and\n"
" including this value will be displayed in a separate column\n");
fprintf(samtools_stderr, " -c add an additional column showing read count\n");
+ fprintf(samtools_stderr, " -H print a comment/header line with column information.\n");
sam_global_opt_help(samtools_stderr, "-.--.--.");
return 1;
}
print_error_errno("bedcov", "can't open BED file '%s'", argv[optind]);
return 2;
}
+
+ if (print_header) {
+ fprintf(samtools_stdout, "#chrom\tstart\tend");
+ for (i = 0; i < n; ++i) {
+ fprintf(samtools_stdout, "\t%s_cov", argv[i+optind+1]);
+ }
+ if (min_depth >= 0) {
+ for (i = 0; i < n; ++i)
+ fprintf(samtools_stdout, "\t%s_depth", argv[i+optind+1]);
+ }
+ if (do_rcount) {
+ for (i = 0; i < n; ++i)
+ fprintf(samtools_stdout, "\t%s_count", argv[i+optind+1]);
+ }
+ fputc('\n', samtools_stdout);
+ }
+
ks = ks_init(fp);
n_plp = calloc(n, sizeof(int));
plp = calloc(n, sizeof(bam_pileup1_t*));
}
mplp = bam_mplp_init(n, read_bam, (void**)aux);
- if (min_depth > DEFAULT_DEPTH)
+ if (min_depth > max_depth)
+ // NB: never happens given current DEFAULT_DEPTH of INT_MAX
bam_mplp_set_maxcnt(mplp, min_depth);
else
- bam_mplp_set_maxcnt(mplp, DEFAULT_DEPTH);
+ bam_mplp_set_maxcnt(mplp, max_depth);
memset(cnt, 0, sizeof(*cnt) * n);
if (min_depth >= 0) memset(pcov, 0, sizeof(*pcov) * n);
/* bedidx.c -- BED file indexing.
Copyright (C) 2011 Broad Institute.
- Copyright (C) 2014, 2017-2019 Genome Research Ltd.
+ Copyright (C) 2014, 2017-2019, 2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk>
hts_pair_pos_t *a;
int *idx;
int filter;
+ hts_pos_t max_idx;
} bed_reglist_t;
#include "htslib/khash.h"
}
#endif
-static int *bed_index_core(int n, hts_pair_pos_t *a)
+static int bed_index_core(bed_reglist_t *regions)
{
- int i, j, l, *idx, *new_idx;
- l = 0; idx = 0;
- for (i = 0; i < n; ++i) {
- hts_pos_t beg, end;
- beg = a[i].beg >> LIDX_SHIFT; end = a[i].end >> LIDX_SHIFT;
- if (l < end + 1) {
- int old_l = l;
- l = end + 1;
- kroundup32(l);
- new_idx = realloc(idx, l * sizeof(*idx));
- if (!new_idx) {
- free(idx);
- return NULL;
- }
- idx = new_idx;
-
- for (j = old_l; j < l; ++j)
- idx[j] = -1;
+ int i, *idx = NULL;
+ size_t idx_size = 0;
+ hts_pos_t last_end = 0;
+ hts_pair_pos_t *a = regions->a;
+
+ // Construct a linear index on regions, to allow rapid lookup of
+ // where to start searching for matches
+ for (i = 0; i < regions->n; ++i) {
+ hts_pos_t beg = a[i].beg >= 0 ? a[i].beg >> LIDX_SHIFT : 0;
+ hts_pos_t end = a[i].end >= 0 ? a[i].end >> LIDX_SHIFT : 0;
+ hts_pos_t j;
+ if (end < last_end)
+ continue; // Can happen for a containment
+ if (end + 1 >= SIZE_MAX / sizeof(*idx)) { // Ensure no overflow
+ errno = ENOMEM;
+ free(idx);
+ return -1;
}
-
- for (j = beg; j < end+1; ++j)
- if (idx[j] < 0)
- idx[j] = i;
+ if (hts_resize(int, (size_t) end + 1, &idx_size, &idx, 0) < 0) {
+ free(idx);
+ return -1;
+ }
+ // Fill any gap prior to this region by pointing to the previous one
+ for (j = last_end; j < beg; j++)
+ idx[j] = i > 0 ? i - 1 : 0;
+ // Fill from max(last_end, beg) to `end` (inclusive) with current region
+ for (; j <= end; j++)
+ idx[j] = i;
+ // Remember where finished for the next gap
+ last_end = end + 1;
}
- return idx;
+ regions->idx = idx;
+ regions->max_idx = last_end;
+ return 0;
}
-static void bed_index(void *_h)
+static int bed_index(reghash_t *h)
{
- reghash_t *h = (reghash_t*)_h;
khint_t k;
for (k = 0; k < kh_end(h); ++k) {
if (kh_exist(h, k)) {
bed_reglist_t *p = &kh_val(h, k);
- if (p->idx) free(p->idx);
+ if (p->idx) {
+ free(p->idx);
+ p->idx = NULL;
+ }
ks_introsort(hts_pair_pos_t, p->n, p->a);
- p->idx = bed_index_core(p->n, p->a);
+ if (bed_index_core(p) != 0) {
+ return -1;
+ }
}
}
+ return 0;
}
-static int bed_minoff(const bed_reglist_t *p, hts_pos_t beg, hts_pos_t end) {
- int i, min_off=0;
-
- if (p && p->idx) {
- min_off = (beg>>LIDX_SHIFT >= p->n)? p->idx[p->n-1] : p->idx[beg>>LIDX_SHIFT];
- if (min_off < 0) { // TODO: this block can be improved, but speed should not matter too much here
- hts_pos_t n = beg>>LIDX_SHIFT;
- if (n > p->n)
- n = p->n;
- for (i = n - 1; i >= 0; --i)
- if (p->idx[i] >= 0)
- break;
- min_off = i >= 0? p->idx[i] : 0;
- }
+static int bed_minoff(const bed_reglist_t *p, hts_pos_t beg) {
+ int min_off=0;
+
+ if (p && p->idx && p->max_idx > 0 && beg >= 0) {
+ min_off = (beg>>LIDX_SHIFT >= p->max_idx)? p->idx[p->max_idx-1] : p->idx[beg>>LIDX_SHIFT];
}
return min_off;
{
int i, min_off;
if (p->n == 0) return 0;
- min_off = bed_minoff(p, beg, end);
+ min_off = bed_minoff(p, beg);
for (i = min_off; i < p->n; ++i) {
if (p->a[i].beg >= end) break; // out of range; no need to proceed
if (NULL == h) return NULL;
// read the list
fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
- if (fp == 0) return 0;
+ if (fp == 0) goto fail;
ks = ks_init(fp);
if (NULL == ks) goto fail; // In case ks_init ever gets error checking...
int ks_len;
fp = NULL;
goto fail;
}
+ if (bed_index(h) != 0)
+ goto fail;
ks_destroy(ks);
free(str.s);
- bed_index(h);
//bed_unify(h);
return h;
fail:
beg = q->a[i].beg;
end = q->a[i].end;
- min_off = bed_minoff(p, beg, end);
+ min_off = bed_minoff(p, beg);
for (j = min_off; j < p->n; ++j) {
if (p->a[j].beg >= end) break; // out of range; no need to proceed
if (p->a[j].end > beg && p->a[j].beg < end) {
//if op==1 insert reg to the bed hash table
if (*op && !(bed_insert(h, reg, beg, end))) {
- fprintf(stderr, "Error when inserting region='%s' in the bed hash table at address=%p!\n", regs[i], h);
+ fprintf(stderr, "Error when inserting region='%s' in the bed hash table at address=%p!\n", regs[i], (void *)h);
}
//if op==0, first insert the regions in the temporary hash table,
//then filter the bed hash table using it
if (!(*op) && !(bed_insert(t, reg, beg, end))) {
- fprintf(stderr, "Error when inserting region='%s' in the temporary hash table at address=%p!\n", regs[i], t);
+ fprintf(stderr, "Error when inserting region='%s' in the temporary hash table at address=%p!\n", regs[i], (void *)t);
}
}
if (!(*op)) {
- bed_index(t);
+ if (bed_index(t) != 0)
+ goto fail;
bed_unify(t);
- h = bed_filter(h, t);
+ if (bed_filter(h, t) == NULL)
+ goto fail;
bed_destroy(t);
+ t = NULL;
}
if (h) {
- bed_index(h);
+ if (bed_index(h) != 0)
+ goto fail;
bed_unify(h);
}
return h;
+
+ fail:
+ // Clean up whichever hash we made
+ if (reg_hash) {
+ if (t)
+ bed_destroy(t);
+ } else {
+ if (h)
+ bed_destroy(h);
+ }
+ return NULL;
}
const char* bed_get(void *reg_hash, int i, int filter) {
/* bedidx.c -- BED file indexing.
Copyright (C) 2011 Broad Institute.
- Copyright (C) 2014, 2017-2019 Genome Research Ltd.
+ Copyright (C) 2014, 2017-2019, 2024 Genome Research Ltd.
Author: Heng Li <lh3@sanger.ac.uk>
hts_pair_pos_t *a;
int *idx;
int filter;
+ hts_pos_t max_idx;
} bed_reglist_t;
#include "htslib/khash.h"
}
#endif
-static int *bed_index_core(int n, hts_pair_pos_t *a)
+static int bed_index_core(bed_reglist_t *regions)
{
- int i, j, l, *idx, *new_idx;
- l = 0; idx = 0;
- for (i = 0; i < n; ++i) {
- hts_pos_t beg, end;
- beg = a[i].beg >> LIDX_SHIFT; end = a[i].end >> LIDX_SHIFT;
- if (l < end + 1) {
- int old_l = l;
- l = end + 1;
- kroundup32(l);
- new_idx = realloc(idx, l * sizeof(*idx));
- if (!new_idx) {
- free(idx);
- return NULL;
- }
- idx = new_idx;
-
- for (j = old_l; j < l; ++j)
- idx[j] = -1;
+ int i, *idx = NULL;
+ size_t idx_size = 0;
+ hts_pos_t last_end = 0;
+ hts_pair_pos_t *a = regions->a;
+
+ // Construct a linear index on regions, to allow rapid lookup of
+ // where to start searching for matches
+ for (i = 0; i < regions->n; ++i) {
+ hts_pos_t beg = a[i].beg >= 0 ? a[i].beg >> LIDX_SHIFT : 0;
+ hts_pos_t end = a[i].end >= 0 ? a[i].end >> LIDX_SHIFT : 0;
+ hts_pos_t j;
+ if (end < last_end)
+ continue; // Can happen for a containment
+ if (end + 1 >= SIZE_MAX / sizeof(*idx)) { // Ensure no overflow
+ errno = ENOMEM;
+ free(idx);
+ return -1;
}
-
- for (j = beg; j < end+1; ++j)
- if (idx[j] < 0)
- idx[j] = i;
+ if (hts_resize(int, (size_t) end + 1, &idx_size, &idx, 0) < 0) {
+ free(idx);
+ return -1;
+ }
+ // Fill any gap prior to this region by pointing to the previous one
+ for (j = last_end; j < beg; j++)
+ idx[j] = i > 0 ? i - 1 : 0;
+ // Fill from max(last_end, beg) to `end` (inclusive) with current region
+ for (; j <= end; j++)
+ idx[j] = i;
+ // Remember where finished for the next gap
+ last_end = end + 1;
}
- return idx;
+ regions->idx = idx;
+ regions->max_idx = last_end;
+ return 0;
}
-static void bed_index(void *_h)
+static int bed_index(reghash_t *h)
{
- reghash_t *h = (reghash_t*)_h;
khint_t k;
for (k = 0; k < kh_end(h); ++k) {
if (kh_exist(h, k)) {
bed_reglist_t *p = &kh_val(h, k);
- if (p->idx) free(p->idx);
+ if (p->idx) {
+ free(p->idx);
+ p->idx = NULL;
+ }
ks_introsort(hts_pair_pos_t, p->n, p->a);
- p->idx = bed_index_core(p->n, p->a);
+ if (bed_index_core(p) != 0) {
+ return -1;
+ }
}
}
+ return 0;
}
-static int bed_minoff(const bed_reglist_t *p, hts_pos_t beg, hts_pos_t end) {
- int i, min_off=0;
-
- if (p && p->idx) {
- min_off = (beg>>LIDX_SHIFT >= p->n)? p->idx[p->n-1] : p->idx[beg>>LIDX_SHIFT];
- if (min_off < 0) { // TODO: this block can be improved, but speed should not matter too much here
- hts_pos_t n = beg>>LIDX_SHIFT;
- if (n > p->n)
- n = p->n;
- for (i = n - 1; i >= 0; --i)
- if (p->idx[i] >= 0)
- break;
- min_off = i >= 0? p->idx[i] : 0;
- }
+static int bed_minoff(const bed_reglist_t *p, hts_pos_t beg) {
+ int min_off=0;
+
+ if (p && p->idx && p->max_idx > 0 && beg >= 0) {
+ min_off = (beg>>LIDX_SHIFT >= p->max_idx)? p->idx[p->max_idx-1] : p->idx[beg>>LIDX_SHIFT];
}
return min_off;
{
int i, min_off;
if (p->n == 0) return 0;
- min_off = bed_minoff(p, beg, end);
+ min_off = bed_minoff(p, beg);
for (i = min_off; i < p->n; ++i) {
if (p->a[i].beg >= end) break; // out of range; no need to proceed
if (NULL == h) return NULL;
// read the list
fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
- if (fp == 0) return 0;
+ if (fp == 0) goto fail;
ks = ks_init(fp);
if (NULL == ks) goto fail; // In case ks_init ever gets error checking...
int ks_len;
fp = NULL;
goto fail;
}
+ if (bed_index(h) != 0)
+ goto fail;
ks_destroy(ks);
free(str.s);
- bed_index(h);
//bed_unify(h);
return h;
fail:
beg = q->a[i].beg;
end = q->a[i].end;
- min_off = bed_minoff(p, beg, end);
+ min_off = bed_minoff(p, beg);
for (j = min_off; j < p->n; ++j) {
if (p->a[j].beg >= end) break; // out of range; no need to proceed
if (p->a[j].end > beg && p->a[j].beg < end) {
//if op==1 insert reg to the bed hash table
if (*op && !(bed_insert(h, reg, beg, end))) {
- fprintf(samtools_stderr, "Error when inserting region='%s' in the bed hash table at address=%p!\n", regs[i], h);
+ fprintf(samtools_stderr, "Error when inserting region='%s' in the bed hash table at address=%p!\n", regs[i], (void *)h);
}
//if op==0, first insert the regions in the temporary hash table,
//then filter the bed hash table using it
if (!(*op) && !(bed_insert(t, reg, beg, end))) {
- fprintf(samtools_stderr, "Error when inserting region='%s' in the temporary hash table at address=%p!\n", regs[i], t);
+ fprintf(samtools_stderr, "Error when inserting region='%s' in the temporary hash table at address=%p!\n", regs[i], (void *)t);
}
}
if (!(*op)) {
- bed_index(t);
+ if (bed_index(t) != 0)
+ goto fail;
bed_unify(t);
- h = bed_filter(h, t);
+ if (bed_filter(h, t) == NULL)
+ goto fail;
bed_destroy(t);
+ t = NULL;
}
if (h) {
- bed_index(h);
+ if (bed_index(h) != 0)
+ goto fail;
bed_unify(h);
}
return h;
+
+ fail:
+ // Clean up whichever hash we made
+ if (reg_hash) {
+ if (t)
+ bed_destroy(t);
+ } else {
+ if (h)
+ bed_destroy(h);
+ }
+ return NULL;
}
const char* bed_get(void *reg_hash, int i, int filter) {
/* coverage.c -- samtools coverage subcommand
Copyright (C) 2018,2019 Florian Breitwieser
- Portions copyright (C) 2019-2021 Genome Research Ltd.
+ Portions copyright (C) 2019-2021, 2023-2024 Genome Research Ltd.
Author: Florian P Breitwieser <florian.bw@gmail.com>
unsigned long long summed_coverage;
unsigned long long summed_baseQ;
unsigned long long summed_mapQ;
+ unsigned long long quality_bases;
unsigned int n_reads;
unsigned int n_selected_reads;
bool covered;
// in bam_plcmd.c
int read_file_list(const char *file_list, int *n, char **argv[]);
-static int usage() {
+static int usage(void) {
fprintf(stdout, "Usage: samtools coverage [options] in1.bam [in2.bam [...]]\n\n"
"Input options:\n"
" -b, --bam-list FILE list of input BAM filenames, one per line\n"
" effectively removing any depth limit.\n"
"Output options:\n"
" -m, --histogram show histogram instead of tabular output\n"
+ " -D, --plot-depth plot depth instead of tabular output\n"
" -A, --ascii show only ASCII characters in histogram\n"
" -o, --output FILE write output to FILE [stdout]\n"
" -H, --no-header don't print a header in tabular mode\n"
stats[tid].n_covered_bases,
100.0 * stats[tid].n_covered_bases / region_len,
stats[tid].summed_coverage / region_len,
- stats[tid].summed_coverage > 0? stats[tid].summed_baseQ/(double) stats[tid].summed_coverage : 0,
+ stats[tid].quality_bases > 0? stats[tid].summed_baseQ/(double) stats[tid].quality_bases : 0,
stats[tid].n_selected_reads > 0? stats[tid].summed_mapQ/(double) stats[tid].n_selected_reads : 0
);
}
void print_hist(FILE *file_out, const sam_hdr_t *h, const stats_aux_t *stats, int tid, const uint32_t *hist,
- const int hist_size, const bool full_utf) {
+ const int hist_size, const bool full_utf, const bool plot_coverage) {
int i, col;
bool show_percentiles = false;
const int n_rows = 10;
double hist_data[hist_size];
double max_val = 0.0;
for (i = 0; i < hist_size; ++i) {
- hist_data[i] = 100 * hist[i] / (double) stats[tid].bin_width;
+ hist_data[i] = (plot_coverage?1:100) * hist[i] / (double) stats[tid].bin_width;
if (hist_data[i] > max_val) max_val = hist_data[i];
}
double row_bin_size = max_val / (double) n_rows;
for (i = n_rows-1; i >= 0; --i) {
double current_bin = row_bin_size * i;
- if (show_percentiles) {
+ if (plot_coverage) {
+ fprintf(file_out, ">%8.1f ",i*row_bin_size);
+ } else if (show_percentiles) {
fprintf(file_out, ">%3i%% ", i*10);
} else {
fprintf(file_out, ">%7.2f%% ", current_bin);
case 5: fprintf(file_out, "Mean coverage: %.3gx",
stats[tid].summed_coverage / region_len); break;
case 4: fprintf(file_out, "Mean baseQ: %.3g",
- stats[tid].summed_baseQ/(double) stats[tid].summed_coverage); break;
+ stats[tid].quality_bases > 0? stats[tid].summed_baseQ/(double) stats[tid].quality_bases : 0); break;
case 3: fprintf(file_out, "Mean mapQ: %.3g",
stats[tid].summed_mapQ/(double) stats[tid].n_selected_reads); break;
case 1: fprintf(file_out, "Histo bin width: %sbp",
readable_bps(stats[tid].bin_width, buf)); break;
- case 0: fprintf(file_out, "Histo max bin: %.5g%%", max_val); break;
+ case 0: if (plot_coverage) {
+ fprintf(file_out, "Histo max cov: %.5g", max_val);
+ } else {
+ fprintf(file_out, "Histo max bin: %.5g%%", max_val);
+ }
+ break;
+
};
fputc('\n', file_out);
}
char **fn = NULL;
int fail_flags = (BAM_FUNMAP | BAM_FSECONDARY | BAM_FQCFAIL | BAM_FDUP); // Default fail flags
int required_flags = 0;
+ int print_value_warning = 0;
int *n_plp = NULL;
sam_hdr_t *h = NULL; // BAM header of the 1st input
bool opt_print_header = true;
bool opt_print_tabular = true;
bool opt_print_histogram = false;
+ bool opt_plot_coverage = false;
bool opt_full_utf = true;
FILE *file_out = stdout;
{"min-bq", required_argument, NULL, 'Q'},
{"histogram", no_argument, NULL, 'm'},
{"ascii", no_argument, NULL, 'A'},
+ {"plot-depth", no_argument, NULL, 'D'},
{"output", required_argument, NULL, 'o'},
{"no-header", no_argument, NULL, 'H'},
{"n-bins", required_argument, NULL, 'w'},
// parse the command line
int c;
opterr = 0;
- while ((c = getopt_long(argc, argv, "Ao:l:q:Q:hHw:r:b:md:", lopts, NULL)) != -1) {
+ while ((c = getopt_long(argc, argv, "Ao:l:q:Q:hHw:r:b:md:D", lopts, NULL)) != -1) {
switch (c) {
case 1:
if ((required_flags = bam_str2flag(optarg)) < 0) {
case 'A': opt_full_utf = false;
opt_print_histogram = true; opt_print_tabular = false;
break;
+ case 'D': opt_print_histogram = true;
+ opt_print_tabular = false;
+ opt_plot_coverage = true;
+ break;
case 'H': opt_print_header = false; break;
case 'h': return usage();
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
status = EXIT_FAILURE;
goto coverage_end;
}
+
while ((ret=bam_mplp_auto(mplp, &tid, &pos, n_plp, plp)) > 0) { // come to the next covered position
if (tid != old_tid) { // Next target sequence
if (old_tid >= 0) {
if (opt_print_histogram) {
- print_hist(file_out, h, stats, old_tid, hist, n_bins, opt_full_utf);
+ print_hist(file_out, h, stats, old_tid, hist, n_bins, opt_full_utf, opt_plot_coverage);
fputc('\n', file_out);
} else if (opt_print_tabular) {
print_tabular_line(file_out, h, stats, old_tid);
for (j = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = plp[i] + j; // DON'T modify plp[][] unless you really know
- if (p->is_del || p->is_refskip) --depth_at_pos; // having dels or refskips at tid:pos
- else if (p->qpos < p->b->core.l_qseq &&
- bam_get_qual(p->b)[p->qpos] < opt_min_baseQ) --depth_at_pos; // low base quality
- else
- stats[tid].summed_baseQ += bam_get_qual(p->b)[p->qpos];
+ if (p->is_del || p->is_refskip) {
+ --depth_at_pos; // having dels or refskips at tid:pos
+ } else if (p->qpos < p->b->core.l_qseq) {
+ if (bam_get_qual(p->b)[p->qpos] < opt_min_baseQ) {
+ --depth_at_pos; // low base quality
+ } else {
+ stats[tid].summed_baseQ += bam_get_qual(p->b)[p->qpos];
+ stats[tid].quality_bases++;
+ }
+ } else {
+ print_value_warning = 1; // no quality at position
+ }
}
+
if (depth_at_pos > 0) {
count_base = true;
stats[tid].summed_coverage += depth_at_pos;
}
- // hist[current_bin] += depth_at_pos; // Add counts to the histogram here to have one based on coverage
- //fprintf(file_out, "\t%d", n_plp[i] - m); // this the depth to output
+
+ if(current_bin < n_bins && opt_plot_coverage) {
+ hist[current_bin] += depth_at_pos;
+ }
}
if (count_base) {
stats[tid].n_covered_bases++;
- if (opt_print_histogram && current_bin < n_bins)
+ if (opt_print_histogram && current_bin < n_bins && !opt_plot_coverage)
++(hist[current_bin]); // Histogram based on breadth of coverage
}
}
if (tid < n_targets && tid >=0) {
if (opt_print_histogram) {
- print_hist(file_out, h, stats, tid, hist, n_bins, opt_full_utf);
+ print_hist(file_out, h, stats, tid, hist, n_bins, opt_full_utf, opt_plot_coverage);
} else if (opt_print_tabular) {
print_tabular_line(file_out, h, stats, tid);
}
}
}
+ if (print_value_warning) {
+ print_error("coverage", "Warning: Missing quality values in alignments. Mean base quality calculated only on available values.");
+ }
+
if (ret < 0) status = EXIT_FAILURE;
coverage_end:
/* coverage.c -- samtools coverage subcommand
Copyright (C) 2018,2019 Florian Breitwieser
- Portions copyright (C) 2019-2021 Genome Research Ltd.
+ Portions copyright (C) 2019-2021, 2023-2024 Genome Research Ltd.
Author: Florian P Breitwieser <florian.bw@gmail.com>
unsigned long long summed_coverage;
unsigned long long summed_baseQ;
unsigned long long summed_mapQ;
+ unsigned long long quality_bases;
unsigned int n_reads;
unsigned int n_selected_reads;
bool covered;
// in bam_plcmd.c
int read_file_list(const char *file_list, int *n, char **argv[]);
-static int usage() {
+static int usage(void) {
fprintf(samtools_stdout, "Usage: samtools coverage [options] in1.bam [in2.bam [...]]\n\n"
"Input options:\n"
" -b, --bam-list FILE list of input BAM filenames, one per line\n"
" effectively removing any depth limit.\n"
"Output options:\n"
" -m, --histogram show histogram instead of tabular output\n"
+ " -D, --plot-depth plot depth instead of tabular output\n"
" -A, --ascii show only ASCII characters in histogram\n"
" -o, --output FILE write output to FILE [samtools_stdout]\n"
" -H, --no-header don't print a header in tabular mode\n"
stats[tid].n_covered_bases,
100.0 * stats[tid].n_covered_bases / region_len,
stats[tid].summed_coverage / region_len,
- stats[tid].summed_coverage > 0? stats[tid].summed_baseQ/(double) stats[tid].summed_coverage : 0,
+ stats[tid].quality_bases > 0? stats[tid].summed_baseQ/(double) stats[tid].quality_bases : 0,
stats[tid].n_selected_reads > 0? stats[tid].summed_mapQ/(double) stats[tid].n_selected_reads : 0
);
}
void print_hist(FILE *file_out, const sam_hdr_t *h, const stats_aux_t *stats, int tid, const uint32_t *hist,
- const int hist_size, const bool full_utf) {
+ const int hist_size, const bool full_utf, const bool plot_coverage) {
int i, col;
bool show_percentiles = false;
const int n_rows = 10;
double hist_data[hist_size];
double max_val = 0.0;
for (i = 0; i < hist_size; ++i) {
- hist_data[i] = 100 * hist[i] / (double) stats[tid].bin_width;
+ hist_data[i] = (plot_coverage?1:100) * hist[i] / (double) stats[tid].bin_width;
if (hist_data[i] > max_val) max_val = hist_data[i];
}
double row_bin_size = max_val / (double) n_rows;
for (i = n_rows-1; i >= 0; --i) {
double current_bin = row_bin_size * i;
- if (show_percentiles) {
+ if (plot_coverage) {
+ fprintf(file_out, ">%8.1f ",i*row_bin_size);
+ } else if (show_percentiles) {
fprintf(file_out, ">%3i%% ", i*10);
} else {
fprintf(file_out, ">%7.2f%% ", current_bin);
case 5: fprintf(file_out, "Mean coverage: %.3gx",
stats[tid].summed_coverage / region_len); break;
case 4: fprintf(file_out, "Mean baseQ: %.3g",
- stats[tid].summed_baseQ/(double) stats[tid].summed_coverage); break;
+ stats[tid].quality_bases > 0? stats[tid].summed_baseQ/(double) stats[tid].quality_bases : 0); break;
case 3: fprintf(file_out, "Mean mapQ: %.3g",
stats[tid].summed_mapQ/(double) stats[tid].n_selected_reads); break;
case 1: fprintf(file_out, "Histo bin width: %sbp",
readable_bps(stats[tid].bin_width, buf)); break;
- case 0: fprintf(file_out, "Histo max bin: %.5g%%", max_val); break;
+ case 0: if (plot_coverage) {
+ fprintf(file_out, "Histo max cov: %.5g", max_val);
+ } else {
+ fprintf(file_out, "Histo max bin: %.5g%%", max_val);
+ }
+ break;
+
};
fputc('\n', file_out);
}
char **fn = NULL;
int fail_flags = (BAM_FUNMAP | BAM_FSECONDARY | BAM_FQCFAIL | BAM_FDUP); // Default fail flags
int required_flags = 0;
+ int print_value_warning = 0;
int *n_plp = NULL;
sam_hdr_t *h = NULL; // BAM header of the 1st input
bool opt_print_header = true;
bool opt_print_tabular = true;
bool opt_print_histogram = false;
+ bool opt_plot_coverage = false;
bool opt_full_utf = true;
FILE *file_out = samtools_stdout;
{"min-bq", required_argument, NULL, 'Q'},
{"histogram", no_argument, NULL, 'm'},
{"ascii", no_argument, NULL, 'A'},
+ {"plot-depth", no_argument, NULL, 'D'},
{"output", required_argument, NULL, 'o'},
{"no-header", no_argument, NULL, 'H'},
{"n-bins", required_argument, NULL, 'w'},
// parse the command line
int c;
opterr = 0;
- while ((c = getopt_long(argc, argv, "Ao:l:q:Q:hHw:r:b:md:", lopts, NULL)) != -1) {
+ while ((c = getopt_long(argc, argv, "Ao:l:q:Q:hHw:r:b:md:D", lopts, NULL)) != -1) {
switch (c) {
case 1:
if ((required_flags = bam_str2flag(optarg)) < 0) {
case 'A': opt_full_utf = false;
opt_print_histogram = true; opt_print_tabular = false;
break;
+ case 'D': opt_print_histogram = true;
+ opt_print_tabular = false;
+ opt_plot_coverage = true;
+ break;
case 'H': opt_print_header = false; break;
case 'h': return usage();
default: if (parse_sam_global_opt(c, optarg, lopts, &ga) == 0) break;
status = EXIT_FAILURE;
goto coverage_end;
}
+
while ((ret=bam_mplp_auto(mplp, &tid, &pos, n_plp, plp)) > 0) { // come to the next covered position
if (tid != old_tid) { // Next target sequence
if (old_tid >= 0) {
if (opt_print_histogram) {
- print_hist(file_out, h, stats, old_tid, hist, n_bins, opt_full_utf);
+ print_hist(file_out, h, stats, old_tid, hist, n_bins, opt_full_utf, opt_plot_coverage);
fputc('\n', file_out);
} else if (opt_print_tabular) {
print_tabular_line(file_out, h, stats, old_tid);
for (j = 0; j < n_plp[i]; ++j) {
const bam_pileup1_t *p = plp[i] + j; // DON'T modify plp[][] unless you really know
- if (p->is_del || p->is_refskip) --depth_at_pos; // having dels or refskips at tid:pos
- else if (p->qpos < p->b->core.l_qseq &&
- bam_get_qual(p->b)[p->qpos] < opt_min_baseQ) --depth_at_pos; // low base quality
- else
- stats[tid].summed_baseQ += bam_get_qual(p->b)[p->qpos];
+ if (p->is_del || p->is_refskip) {
+ --depth_at_pos; // having dels or refskips at tid:pos
+ } else if (p->qpos < p->b->core.l_qseq) {
+ if (bam_get_qual(p->b)[p->qpos] < opt_min_baseQ) {
+ --depth_at_pos; // low base quality
+ } else {
+ stats[tid].summed_baseQ += bam_get_qual(p->b)[p->qpos];
+ stats[tid].quality_bases++;
+ }
+ } else {
+ print_value_warning = 1; // no quality at position
+ }
}
+
if (depth_at_pos > 0) {
count_base = true;
stats[tid].summed_coverage += depth_at_pos;
}
- // hist[current_bin] += depth_at_pos; // Add counts to the histogram here to have one based on coverage
- //fprintf(file_out, "\t%d", n_plp[i] - m); // this the depth to output
+
+ if(current_bin < n_bins && opt_plot_coverage) {
+ hist[current_bin] += depth_at_pos;
+ }
}
if (count_base) {
stats[tid].n_covered_bases++;
- if (opt_print_histogram && current_bin < n_bins)
+ if (opt_print_histogram && current_bin < n_bins && !opt_plot_coverage)
++(hist[current_bin]); // Histogram based on breadth of coverage
}
}
if (tid < n_targets && tid >=0) {
if (opt_print_histogram) {
- print_hist(file_out, h, stats, tid, hist, n_bins, opt_full_utf);
+ print_hist(file_out, h, stats, tid, hist, n_bins, opt_full_utf, opt_plot_coverage);
} else if (opt_print_tabular) {
print_tabular_line(file_out, h, stats, tid);
}
}
}
+ if (print_value_warning) {
+ print_error("coverage", "Warning: Missing quality values in alignments. Mean base quality calculated only on available values.");
+ }
+
if (ret < 0) status = EXIT_FAILURE;
coverage_end:
/* faidx.c -- faidx subcommand.
- Copyright (C) 2008, 2009, 2013, 2016, 2018-2020, 2022 Genome Research Ltd.
+ Copyright (C) 2008, 2009, 2013, 2016, 2018-2020, 2022, 2024 Genome Research Ltd.
Portions copyright (C) 2011 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
#include <htslib/hts.h>
#include <htslib/hfile.h>
#include <htslib/kstring.h>
+#include <htslib/bgzf.h>
+#include <htslib/thread_pool.h>
+#include "sam_opts.h"
#include "samtools.h"
// Negative indicates the same as input data
# define ABS(x) ((x)>=0?(x):-(x))
#endif
+//new params required for output creation
+typedef struct output {
+ int isbgzip; //is bgzip or uncompressed file
+ FILE *fp; //uncompressed file pointer
+ BGZF *bgzf_fp; //bgzf file pointer
+ sam_global_args *gopt; //options
+ kstring_t buffer;
+} output;
+
static unsigned char comp_base[256] = {
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
}
}
+/// wrappedwrite - wraps the fwrite and bgzf_write
+/** @param out - pointer to data required to write output
+* @param buffer - data to write
+* @param length - data length
+* returns error or length written on success
+*/
+static inline size_t wrappedwrite(output *out, const char *buffer, size_t length)
+{
+ return out->isbgzip ? bgzf_write(out->bgzf_fp, buffer, length) :
+ fwrite(buffer, 1, length, out->fp);
+}
-static int write_line(faidx_t *faid, FILE *file, const char *line, const char *name,
+static int write_line(faidx_t *faid, output *out, const char *line, const char *name,
const int ignore, const hts_pos_t length, const hts_pos_t seq_len) {
int id;
hts_pos_t beg, end;
for (i = 0; i < seq_sz; i += length)
{
hts_pos_t len = i + length < seq_sz ? length : seq_sz - i;
- if (fwrite(line + i, 1, len, file) < len ||
- fputc('\n', file) == EOF) {
+ if (wrappedwrite(out, line + i, len) < len ||
+ wrappedwrite(out, "\n", 1) < 1) {
print_error_errno("faidx", "failed to write output");
return EXIT_FAILURE;
}
}
-static int write_output(faidx_t *faid, FILE *file, const char *name, const int ignore,
+static int write_output(faidx_t *faid, output *out, const char *name, const int ignore,
const hts_pos_t length, const int rev,
const char *pos_strand_name, const char *neg_strand_name,
enum fai_format_options format) {
- hts_pos_t seq_len, wrap_len = length;
+ hts_pos_t seq_len, wrap_len = length, len = 0;
+ char *seq = NULL, *qual = NULL;
+ int ret = EXIT_FAILURE;
+
if (wrap_len < 0)
wrap_len = fai_line_length(faid, name);
if (wrap_len <= 0)
wrap_len = HTS_POS_MAX;
- char *seq = fai_fetch64(faid, name, &seq_len);
-
- if (format == FAI_FASTA) {
- fprintf(file, ">%s%s\n", name, rev ? neg_strand_name : pos_strand_name);
- } else {
- fprintf(file, "@%s%s\n", name, rev ? neg_strand_name : pos_strand_name);
- }
+ seq = fai_fetch64(faid, name, &seq_len);
if (rev && seq_len > 0) {
reverse_complement(seq, seq_len);
}
-
- if (write_line(faid, file, seq, name, ignore, wrap_len, seq_len)
- == EXIT_FAILURE) {
- free(seq);
- return EXIT_FAILURE;
+ //write the name
+ len = ksprintf(&out->buffer, "%c%s%s\n", format == FAI_FASTA ? '>' : '@', name, rev ? neg_strand_name : pos_strand_name);
+ if (wrappedwrite(out, out->buffer.s, out->buffer.l) < len) {
+ fprintf(stderr,"[faidx] Failed to write buffer\n");
+ goto exit;
+ }
+ ks_clear(&out->buffer);
+ //write bases
+ if ((ret = write_line(faid, out, seq, name, ignore, wrap_len, seq_len) == EXIT_FAILURE)) {
+ goto exit;
}
-
- free(seq);
if (format == FAI_FASTQ) {
- fprintf(file, "+\n");
-
- char *qual = fai_fetchqual64(faid, name, &seq_len);
-
+ //write quality
+ qual = fai_fetchqual64(faid, name, &seq_len);
if (rev && seq_len > 0) {
reverse(qual, seq_len);
}
- if (write_line(faid, file, qual, name, ignore, wrap_len, seq_len)
- == EXIT_FAILURE) {
- free(qual);
- return EXIT_FAILURE;
+ len = ksprintf(&out->buffer, "+\n");
+ if (wrappedwrite(out, out->buffer.s, out->buffer.l) < len) {
+ fprintf(stderr,"[faidx] Failed to write buffer\n");
+ goto exit;
}
+ ks_clear(&out->buffer);
+ if ((ret = write_line(faid, out, qual, name, ignore, wrap_len, seq_len) == EXIT_FAILURE)) {
+ goto exit;
+ }
+ }
+ ret = EXIT_SUCCESS;
+exit:
+ if (seq) {
+ free(seq);
+ }
+ if (qual) {
free(qual);
}
-
- return EXIT_SUCCESS;
+ return ret;
}
-static int read_regions_from_file(faidx_t *faid, hFILE *in_file, FILE *file, const int ignore,
+static int read_regions_from_file(faidx_t *faid, hFILE *in_file, output *out, const int ignore,
const hts_pos_t length, const int rev,
const char *pos_strand_name,
const char *neg_strand_name,
int ret = EXIT_FAILURE;
while (line.l = 0, kgetline(&line, (kgets_func *)hgets, in_file) >= 0) {
- if ((ret = write_output(faid, file, line.s, ignore, length, rev, pos_strand_name, neg_strand_name, format)) == EXIT_FAILURE) {
+ if ((ret = write_output(faid, out, line.s, ignore, length, rev, pos_strand_name, neg_strand_name, format)) == EXIT_FAILURE) {
break;
}
}
fprintf(fp, "Usage: samtools %s [<reg> [...]]\n", tool);
fprintf(fp, "Option: \n"
- " -o, --output FILE Write %s to file.\n"
- " -n, --length INT Length of %s sequence line. [60]\n"
- " -c, --continue Continue after trying to retrieve missing region.\n"
- " -r, --region-file FILE File of regions. Format is chr:from-to. One per line.\n"
- " -i, --reverse-complement Reverse complement sequences.\n"
- " --mark-strand TYPE Add strand indicator to sequence name\n"
- " TYPE = rc for /rc on negative strand (default)\n"
- " no for no strand indicator\n"
- " sign for (+) / (-)\n"
- " custom,<pos>,<neg> for custom indicator\n"
- " --fai-idx FILE name of the index file (default %s.fai).\n"
- " --gzi-idx FILE name of compressed file index (default %s.gz.gzi).\n",
+ " -o, --output FILE Write %s to file.\n"
+ " -n, --length INT Length of %s sequence line. [60]\n"
+ " -c, --continue Continue after trying to retrieve missing region.\n"
+ " -r, --region-file FILE File of regions. Format is chr:from-to. One per line.\n"
+ " -i, --reverse-complement Reverse complement sequences.\n"
+ " --mark-strand TYPE Add strand indicator to sequence name\n"
+ " TYPE = rc for /rc on negative strand (default)\n"
+ " no for no strand indicator\n"
+ " sign for (+) / (-)\n"
+ "
custom,<pos>,<neg> for custom indicator\n"
+ " --fai-idx FILE name of the index file (default %s.fai).\n"
+ " --gzi-idx FILE name of compressed file index (default %s.gz.gzi).\n",
file_type, file_type, index_name, index_name);
if (format == FAI_FASTA) {
- fprintf(fp, " -f, --fastq File and index in FASTQ format.\n");
+ fprintf(fp, " -f, --fastq File and index in FASTQ format.\n");
}
- fprintf(fp, " -h, --help This message.\n");
+ fprintf(fp, " -h, --help This message.\n");
+ sam_global_opt_help(fp, "---.-@--");
return exit_status;
}
char *strand_names = NULL; // Used for custom strand annotation
char *fai_name = NULL; // specified index name
char *gzi_name = NULL; // specified compressed index name
- FILE* file_out = stdout;/* output stream */
+ sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
+ int exit_status = EXIT_FAILURE, flushed = 0;
+ struct output out = { 0, stdout, NULL, &ga, KS_INITIALIZE}; //data required for output writing
+ faidx_t *fai = NULL;
+ hts_tpool *pool = NULL;
static const struct option lopts[] = {
+ SAM_OPT_GLOBAL_OPTIONS('-', '-', '-', 0, '-', '@'), //output format opt and thread count - long options
{ "output", required_argument, NULL, 'o' },
{ "help", no_argument, NULL, 'h' },
{ "length", required_argument, NULL, 'n' },
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "ho:n:cr:fi", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "ho:n:cr:fi@:", lopts, NULL)) >= 0) {
switch (c) {
- case 'o': output_file = optarg; break;
+ case 'o':
+ {
+ output_file = optarg;
+ char *ext = strrchr(output_file, '.');
+ if (!ext) {
+ break;
+ }
+ if (!strcmp(ext, ".gz") || !strcmp(ext, ".bgz") || !strcmp(ext, ".bgzf")) {
+ out.isbgzip = 1; //bgzip output
+ }
+ break;
+ }
case 'n': line_len = strtol(optarg, NULL, 10);
if (line_len < 0) {
fprintf(stderr,"[faidx] bad line length '%s', using default:%d\n",optarg,ABS(DEFAULT_FASTA_LINE_LEN));
case 'r': region_file = optarg; break;
case 'f': format = FAI_FASTQ; break;
case 'i': rev = 1; break;
- case '?': return usage(stderr, format, EXIT_FAILURE);
- case 'h': return usage(stdout, format, EXIT_SUCCESS);
+ case '?':
+ exit_status = usage(stderr, format, EXIT_FAILURE);
+ goto exit2;
+ break;
+ case 'h':
+ exit_status = usage(stdout, format, EXIT_SUCCESS);
+ goto exit2;
+ break;
case 1000:
if (strcmp(optarg, "no") == 0) {
pos_strand_name = neg_strand_name = "";
strand_names = pos_strand_name = malloc(len + 2);
if (!strand_names) {
fprintf(stderr, "[faidx] Out of memory\n");
- return EXIT_FAILURE;
+ goto exit2;
}
neg_strand_name = pos_strand_name + comma + 1;
memcpy(pos_strand_name, optarg + 7, comma);
neg_strand_name[len - comma] = '\0';
} else {
fprintf(stderr, "[faidx] Unknown --mark-strand option \"%s\"\n", optarg);
- return usage(stderr, format, EXIT_FAILURE);
+ exit_status = usage(stderr, format, EXIT_FAILURE);
+ goto exit2;
}
break;
case 1001: fai_name = optarg; break;
case 1002: gzi_name = optarg; break;
- default: break;
+ // handle standard samtools options like thread count, compression level...
+ default:
+ if (parse_sam_global_opt(c, optarg, lopts, &ga)) {
+ fprintf(stderr, "[faidx] Invalid option \"%s\"\n", optarg);
+ goto exit2;
+ }
+ break;
}
}
- if ( argc==optind )
- return usage(stdout, format, EXIT_SUCCESS);
+ if ( argc==optind ) {
+ exit_status = usage(stdout, format, EXIT_SUCCESS);
+ goto exit2;
+ }
if (optind+1 == argc && !region_file) {
if (output_file && !fai_name)
else
fprintf(stderr, "\n");
- return EXIT_FAILURE;
+ goto exit2;
}
+ exit_status = EXIT_SUCCESS;
+ goto exit2;
+ }
- return 0;
+ if (out.gopt->nthreads > 0) { //setup thread pool
+ if (!(pool = hts_tpool_init(out.gopt->nthreads))) {
+ fprintf(stderr, "Failed to setup thread pool\n");
+ }
}
- faidx_t *fai = fai_load3_format(argv[optind], fai_name, gzi_name, FAI_CREATE, format);
+ fai = fai_load3_format(argv[optind], fai_name, gzi_name, FAI_CREATE, format);
if (!fai) {
if (fai_name)
fprintf(stderr, "[faidx] Could not load fai index %s", fai_name);
else
- fprintf(stderr, "[faidx] Could not build fai index %s.fai", argv[optind]);
+ fprintf(stderr, "[faidx] Could not load fai index %s.fai", argv[optind]);
if (gzi_name)
fprintf(stderr, " or compressed index %s\n", gzi_name);
else
fprintf(stderr, "\n");
- return EXIT_FAILURE;
+ goto exit2;
+ }
+
+ if (pool) { //use thread pool if set
+ if (fai_thread_pool(fai, pool, 0)) {
+ fprintf(stderr, "Failed to set thread pool for reading\n");
+ }
}
/** output file provided by user */
if( output_file != NULL ) {
if( strcmp( output_file, argv[optind] ) == 0 ) {
fprintf(stderr,"[faidx] Same input/output : %s\n", output_file);
- return EXIT_FAILURE;
+ goto exit2;
+ }
+ if (!out.isbgzip) {
+ out.fp = fopen( output_file, "w" );
+ } else {
+ hts_opt *opts = (hts_opt *)(out.gopt->out.specific);
+ char mode[13] = "w";
+ int level = 4; //default compression level
+ while (opts) {
+ if (opts->opt == HTS_OPT_COMPRESSION_LEVEL) { //compression level
+ level = opts->val.i;
+ break;
+ }
+ opts = opts->next;
+ }
+ if (level >= 0) {
+ snprintf(mode, sizeof(mode), "w%d", level); //pass compression with mode
+ }
+ out.bgzf_fp = bgzf_open(output_file, mode);
}
- file_out = fopen( output_file, "w" );
-
- if( file_out == NULL) {
+ if( (!out.isbgzip && out.fp == NULL) || (out.isbgzip && out.bgzf_fp == NULL)) {
fprintf(stderr,"[faidx] Cannot open \"%s\" for writing :%s.\n", output_file, strerror(errno) );
- return EXIT_FAILURE;
+ goto exit2;
+ }
+ if (out.isbgzip && pool) { //use thread pool if set
+ if (bgzf_thread_pool(out.bgzf_fp, pool, 0)) {
+ fprintf(stderr, "Failed to set thread pool for writing\n");
+ }
}
}
- int exit_status = EXIT_SUCCESS;
-
if (region_file) {
hFILE *rf;
if ((rf = hopen(region_file, "r"))) {
- exit_status = read_regions_from_file(fai, rf, file_out, ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
+ exit_status = read_regions_from_file(fai, rf, &out, ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
if (hclose(rf) != 0) {
fprintf(stderr, "[faidx] Warning: failed to close %s", region_file);
}
+ if (exit_status == EXIT_FAILURE) {
+ goto exit1;
+ }
} else {
fprintf(stderr, "[faidx] Failed to open \"%s\" for reading.\n", region_file);
- exit_status = EXIT_FAILURE;
+ goto exit1;
}
}
+ exit_status = EXIT_SUCCESS;
while ( ++optind<argc && exit_status == EXIT_SUCCESS) {
- exit_status = write_output(fai, file_out, argv[optind], ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
+ exit_status = write_output(fai, &out, argv[optind], ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
}
- fai_destroy(fai);
-
- if (fflush(file_out) == EOF) {
- print_error_errno("faidx", "failed to flush output");
+ flushed = out.isbgzip ? bgzf_flush(out.bgzf_fp) : fflush(out.fp);
+ if (flushed == EOF) {
+ print_error_errno("faidx", "Failed to flush output\n");
exit_status = EXIT_FAILURE;
}
- if( output_file != NULL) fclose(file_out);
- free(strand_names);
+exit1:
+ if( output_file != NULL && !out.isbgzip) {
+ fclose(out.fp); //no need to check result as already flushed
+ } else if( output_file != NULL && out.isbgzip) {
+ if (bgzf_close(out.bgzf_fp) < 0) {
+ print_error_errno("faidx", "Failed to close output\n");
+ exit_status = EXIT_FAILURE;
+ }
+ }
+
+exit2:
+ if (strand_names) {
+ free(strand_names);
+ }
+ if (fai) {
+ fai_destroy(fai);
+ }
+ if (pool) {
+ hts_tpool_destroy(pool);
+ }
+ sam_global_args_free(&ga);
+ ks_free(&out.buffer);
return exit_status;
}
/* faidx.c -- faidx subcommand.
- Copyright (C) 2008, 2009, 2013, 2016, 2018-2020, 2022 Genome Research Ltd.
+ Copyright (C) 2008, 2009, 2013, 2016, 2018-2020, 2022, 2024 Genome Research Ltd.
Portions copyright (C) 2011 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
#include <htslib/hts.h>
#include <htslib/hfile.h>
#include <htslib/kstring.h>
+#include <htslib/bgzf.h>
+#include <htslib/thread_pool.h>
+#include "sam_opts.h"
#include "samtools.h"
// Negative indicates the same as input data
# define ABS(x) ((x)>=0?(x):-(x))
#endif
+//new params required for output creation
+typedef struct output {
+ int isbgzip; //is bgzip or uncompressed file
+ FILE *fp; //uncompressed file pointer
+ BGZF *bgzf_fp; //bgzf file pointer
+ sam_global_args *gopt; //options
+ kstring_t buffer;
+} output;
+
static unsigned char comp_base[256] = {
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
}
}
+/// wrappedwrite - wraps the fwrite and bgzf_write
+/** @param out - pointer to data required to write output
+* @param buffer - data to write
+* @param length - data length
+* returns error or length written on success
+*/
+static inline size_t wrappedwrite(output *out, const char *buffer, size_t length)
+{
+ return out->isbgzip ? bgzf_write(out->bgzf_fp, buffer, length) :
+ fwrite(buffer, 1, length, out->fp);
+}
-static int write_line(faidx_t *faid, FILE *file, const char *line, const char *name,
+static int write_line(faidx_t *faid, output *out, const char *line, const char *name,
const int ignore, const hts_pos_t length, const hts_pos_t seq_len) {
int id;
hts_pos_t beg, end;
for (i = 0; i < seq_sz; i += length)
{
hts_pos_t len = i + length < seq_sz ? length : seq_sz - i;
- if (fwrite(line + i, 1, len, file) < len ||
- fputc('\n', file) == EOF) {
+ if (wrappedwrite(out, line + i, len) < len ||
+ wrappedwrite(out, "\n", 1) < 1) {
print_error_errno("faidx", "failed to write output");
return EXIT_FAILURE;
}
}
-static int write_output(faidx_t *faid, FILE *file, const char *name, const int ignore,
+static int write_output(faidx_t *faid, output *out, const char *name, const int ignore,
const hts_pos_t length, const int rev,
const char *pos_strand_name, const char *neg_strand_name,
enum fai_format_options format) {
- hts_pos_t seq_len, wrap_len = length;
+ hts_pos_t seq_len, wrap_len = length, len = 0;
+ char *seq = NULL, *qual = NULL;
+ int ret = EXIT_FAILURE;
+
if (wrap_len < 0)
wrap_len = fai_line_length(faid, name);
if (wrap_len <= 0)
wrap_len = HTS_POS_MAX;
- char *seq = fai_fetch64(faid, name, &seq_len);
-
- if (format == FAI_FASTA) {
- fprintf(file, ">%s%s\n", name, rev ? neg_strand_name : pos_strand_name);
- } else {
- fprintf(file, "@%s%s\n", name, rev ? neg_strand_name : pos_strand_name);
- }
+ seq = fai_fetch64(faid, name, &seq_len);
if (rev && seq_len > 0) {
reverse_complement(seq, seq_len);
}
-
- if (write_line(faid, file, seq, name, ignore, wrap_len, seq_len)
- == EXIT_FAILURE) {
- free(seq);
- return EXIT_FAILURE;
+ //write the name
+ len = ksprintf(&out->buffer, "%c%s%s\n", format == FAI_FASTA ? '>' : '@', name, rev ? neg_strand_name : pos_strand_name);
+ if (wrappedwrite(out, out->buffer.s, out->buffer.l) < len) {
+ fprintf(samtools_stderr,"[faidx] Failed to write buffer\n");
+ goto exit;
+ }
+ ks_clear(&out->buffer);
+ //write bases
+ if ((ret = write_line(faid, out, seq, name, ignore, wrap_len, seq_len) == EXIT_FAILURE)) {
+ goto exit;
}
-
- free(seq);
if (format == FAI_FASTQ) {
- fprintf(file, "+\n");
-
- char *qual = fai_fetchqual64(faid, name, &seq_len);
-
+ //write quality
+ qual = fai_fetchqual64(faid, name, &seq_len);
if (rev && seq_len > 0) {
reverse(qual, seq_len);
}
- if (write_line(faid, file, qual, name, ignore, wrap_len, seq_len)
- == EXIT_FAILURE) {
- free(qual);
- return EXIT_FAILURE;
+ len = ksprintf(&out->buffer, "+\n");
+ if (wrappedwrite(out, out->buffer.s, out->buffer.l) < len) {
+ fprintf(samtools_stderr,"[faidx] Failed to write buffer\n");
+ goto exit;
}
+ ks_clear(&out->buffer);
+ if ((ret = write_line(faid, out, qual, name, ignore, wrap_len, seq_len) == EXIT_FAILURE)) {
+ goto exit;
+ }
+ }
+ ret = EXIT_SUCCESS;
+exit:
+ if (seq) {
+ free(seq);
+ }
+ if (qual) {
free(qual);
}
-
- return EXIT_SUCCESS;
+ return ret;
}
-static int read_regions_from_file(faidx_t *faid, hFILE *in_file, FILE *file, const int ignore,
+static int read_regions_from_file(faidx_t *faid, hFILE *in_file, output *out, const int ignore,
const hts_pos_t length, const int rev,
const char *pos_strand_name,
const char *neg_strand_name,
int ret = EXIT_FAILURE;
while (line.l = 0, kgetline(&line, (kgets_func *)hgets, in_file) >= 0) {
- if ((ret = write_output(faid, file, line.s, ignore, length, rev, pos_strand_name, neg_strand_name, format)) == EXIT_FAILURE) {
+ if ((ret = write_output(faid, out, line.s, ignore, length, rev, pos_strand_name, neg_strand_name, format)) == EXIT_FAILURE) {
break;
}
}
fprintf(fp, "Usage: samtools %s [<reg> [...]]\n", tool);
fprintf(fp, "Option: \n"
- " -o, --output FILE Write %s to file.\n"
- " -n, --length INT Length of %s sequence line. [60]\n"
- " -c, --continue Continue after trying to retrieve missing region.\n"
- " -r, --region-file FILE File of regions. Format is chr:from-to. One per line.\n"
- " -i, --reverse-complement Reverse complement sequences.\n"
- " --mark-strand TYPE Add strand indicator to sequence name\n"
- " TYPE = rc for /rc on negative strand (default)\n"
- " no for no strand indicator\n"
- " sign for (+) / (-)\n"
- " custom,<pos>,<neg> for custom indicator\n"
- " --fai-idx FILE name of the index file (default %s.fai).\n"
- " --gzi-idx FILE name of compressed file index (default %s.gz.gzi).\n",
+ " -o, --output FILE Write %s to file.\n"
+ " -n, --length INT Length of %s sequence line. [60]\n"
+ " -c, --continue Continue after trying to retrieve missing region.\n"
+ " -r, --region-file FILE File of regions. Format is chr:from-to. One per line.\n"
+ " -i, --reverse-complement Reverse complement sequences.\n"
+ " --mark-strand TYPE Add strand indicator to sequence name\n"
+ " TYPE = rc for /rc on negative strand (default)\n"
+ " no for no strand indicator\n"
+ " sign for (+) / (-)\n"
+ "
custom,<pos>,<neg> for custom indicator\n"
+ " --fai-idx FILE name of the index file (default %s.fai).\n"
+ " --gzi-idx FILE name of compressed file index (default %s.gz.gzi).\n",
file_type, file_type, index_name, index_name);
if (format == FAI_FASTA) {
- fprintf(fp, " -f, --fastq File and index in FASTQ format.\n");
+ fprintf(fp, " -f, --fastq File and index in FASTQ format.\n");
}
- fprintf(fp, " -h, --help This message.\n");
+ fprintf(fp, " -h, --help This message.\n");
+ sam_global_opt_help(fp, "---.-@--");
return exit_status;
}
char *strand_names = NULL; // Used for custom strand annotation
char *fai_name = NULL; // specified index name
char *gzi_name = NULL; // specified compressed index name
- FILE* file_out = samtools_stdout;/* output stream */
+ sam_global_args ga = SAM_GLOBAL_ARGS_INIT;
+ int exit_status = EXIT_FAILURE, flushed = 0;
+ struct output out = { 0, samtools_stdout, NULL, &ga, KS_INITIALIZE}; //data required for output writing
+ faidx_t *fai = NULL;
+ hts_tpool *pool = NULL;
static const struct option lopts[] = {
+ SAM_OPT_GLOBAL_OPTIONS('-', '-', '-', 0, '-', '@'), //output format opt and thread count - long options
{ "output", required_argument, NULL, 'o' },
{ "help", no_argument, NULL, 'h' },
{ "length", required_argument, NULL, 'n' },
{ NULL, 0, NULL, 0 }
};
- while ((c = getopt_long(argc, argv, "ho:n:cr:fi", lopts, NULL)) >= 0) {
+ while ((c = getopt_long(argc, argv, "ho:n:cr:fi@:", lopts, NULL)) >= 0) {
switch (c) {
- case 'o': output_file = optarg; break;
+ case 'o':
+ {
+ output_file = optarg;
+ char *ext = strrchr(output_file, '.');
+ if (!ext) {
+ break;
+ }
+ if (!strcmp(ext, ".gz") || !strcmp(ext, ".bgz") || !strcmp(ext, ".bgzf")) {
+ out.isbgzip = 1; //bgzip output
+ }
+ break;
+ }
case 'n': line_len = strtol(optarg, NULL, 10);
if (line_len < 0) {
fprintf(samtools_stderr,"[faidx] bad line length '%s', using default:%d\n",optarg,ABS(DEFAULT_FASTA_LINE_LEN));
case 'r': region_file = optarg; break;
case 'f': format = FAI_FASTQ; break;
case 'i': rev = 1; break;
- case '?': return usage(samtools_stderr, format, EXIT_FAILURE);
- case 'h': return usage(samtools_stdout, format, EXIT_SUCCESS);
+ case '?':
+ exit_status = usage(samtools_stderr, format, EXIT_FAILURE);
+ goto exit2;
+ break;
+ case 'h':
+ exit_status = usage(samtools_stdout, format, EXIT_SUCCESS);
+ goto exit2;
+ break;
case 1000:
if (strcmp(optarg, "no") == 0) {
pos_strand_name = neg_strand_name = "";
strand_names = pos_strand_name = malloc(len + 2);
if (!strand_names) {
fprintf(samtools_stderr, "[faidx] Out of memory\n");
- return EXIT_FAILURE;
+ goto exit2;
}
neg_strand_name = pos_strand_name + comma + 1;
memcpy(pos_strand_name, optarg + 7, comma);
neg_strand_name[len - comma] = '\0';
} else {
fprintf(samtools_stderr, "[faidx] Unknown --mark-strand option \"%s\"\n", optarg);
- return usage(samtools_stderr, format, EXIT_FAILURE);
+ exit_status = usage(samtools_stderr, format, EXIT_FAILURE);
+ goto exit2;
}
break;
case 1001: fai_name = optarg; break;
case 1002: gzi_name = optarg; break;
- default: break;
+ // handle standard samtools options like thread count, compression level...
+ default:
+ if (parse_sam_global_opt(c, optarg, lopts, &ga)) {
+ fprintf(samtools_stderr, "[faidx] Invalid option \"%s\"\n", optarg);
+ goto exit2;
+ }
+ break;
}
}
- if ( argc==optind )
- return usage(samtools_stdout, format, EXIT_SUCCESS);
+ if ( argc==optind ) {
+ exit_status = usage(samtools_stdout, format, EXIT_SUCCESS);
+ goto exit2;
+ }
if (optind+1 == argc && !region_file) {
if (output_file && !fai_name)
else
fprintf(samtools_stderr, "\n");
- return EXIT_FAILURE;
+ goto exit2;
}
+ exit_status = EXIT_SUCCESS;
+ goto exit2;
+ }
- return 0;
+ if (out.gopt->nthreads > 0) { //setup thread pool
+ if (!(pool = hts_tpool_init(out.gopt->nthreads))) {
+ fprintf(samtools_stderr, "Failed to setup thread pool\n");
+ }
}
- faidx_t *fai = fai_load3_format(argv[optind], fai_name, gzi_name, FAI_CREATE, format);
+ fai = fai_load3_format(argv[optind], fai_name, gzi_name, FAI_CREATE, format);
if (!fai) {
if (fai_name)
fprintf(samtools_stderr, "[faidx] Could not load fai index %s", fai_name);
else
- fprintf(samtools_stderr, "[faidx] Could not build fai index %s.fai", argv[optind]);
+ fprintf(samtools_stderr, "[faidx] Could not load fai index %s.fai", argv[optind]);
if (gzi_name)
fprintf(samtools_stderr, " or compressed index %s\n", gzi_name);
else
fprintf(samtools_stderr, "\n");
- return EXIT_FAILURE;
+ goto exit2;
+ }
+
+ if (pool) { //use thread pool if set
+ if (fai_thread_pool(fai, pool, 0)) {
+ fprintf(samtools_stderr, "Failed to set thread pool for reading\n");
+ }
}
/** output file provided by user */
if( output_file != NULL ) {
if( strcmp( output_file, argv[optind] ) == 0 ) {
fprintf(samtools_stderr,"[faidx] Same input/output : %s\n", output_file);
- return EXIT_FAILURE;
+ goto exit2;
+ }
+ if (!out.isbgzip) {
+ out.fp = fopen( output_file, "w" );
+ } else {
+ hts_opt *opts = (hts_opt *)(out.gopt->out.specific);
+ char mode[13] = "w";
+ int level = 4; //default compression level
+ while (opts) {
+ if (opts->opt == HTS_OPT_COMPRESSION_LEVEL) { //compression level
+ level = opts->val.i;
+ break;
+ }
+ opts = opts->next;
+ }
+ if (level >= 0) {
+ snprintf(mode, sizeof(mode), "w%d", level); //pass compression with mode
+ }
+ out.bgzf_fp = bgzf_open(output_file, mode);
}
- file_out = fopen( output_file, "w" );
-
- if( file_out == NULL) {
+ if( (!out.isbgzip && out.fp == NULL) || (out.isbgzip && out.bgzf_fp == NULL)) {
fprintf(samtools_stderr,"[faidx] Cannot open \"%s\" for writing :%s.\n", output_file, strerror(errno) );
- return EXIT_FAILURE;
+ goto exit2;
+ }
+ if (out.isbgzip && pool) { //use thread pool if set
+ if (bgzf_thread_pool(out.bgzf_fp, pool, 0)) {
+ fprintf(samtools_stderr, "Failed to set thread pool for writing\n");
+ }
}
}
- int exit_status = EXIT_SUCCESS;
-
if (region_file) {
hFILE *rf;
if ((rf = hopen(region_file, "r"))) {
- exit_status = read_regions_from_file(fai, rf, file_out, ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
+ exit_status = read_regions_from_file(fai, rf, &out, ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
if (hclose(rf) != 0) {
fprintf(samtools_stderr, "[faidx] Warning: failed to close %s", region_file);
}
+ if (exit_status == EXIT_FAILURE) {
+ goto exit1;
+ }
} else {
fprintf(samtools_stderr, "[faidx] Failed to open \"%s\" for reading.\n", region_file);
- exit_status = EXIT_FAILURE;
+ goto exit1;
}
}
+ exit_status = EXIT_SUCCESS;
while ( ++optind<argc && exit_status == EXIT_SUCCESS) {
- exit_status = write_output(fai, file_out, argv[optind], ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
+ exit_status = write_output(fai, &out, argv[optind], ignore_error, line_len, rev, pos_strand_name, neg_strand_name, format);
}
- fai_destroy(fai);
-
- if (fflush(file_out) == EOF) {
- print_error_errno("faidx", "failed to flush output");
+ flushed = out.isbgzip ? bgzf_flush(out.bgzf_fp) : fflush(out.fp);
+ if (flushed == EOF) {
+ print_error_errno("faidx", "Failed to flush output\n");
exit_status = EXIT_FAILURE;
}
- if( output_file != NULL) fclose(file_out);
- free(strand_names);
+exit1:
+ if( output_file != NULL && !out.isbgzip) {
+ fclose(out.fp); //no need to check result as already flushed
+ } else if( output_file != NULL && out.isbgzip) {
+ if (bgzf_close(out.bgzf_fp) < 0) {
+ print_error_errno("faidx", "Failed to close output\n");
+ exit_status = EXIT_FAILURE;
+ }
+ }
+
+exit2:
+ if (strand_names) {
+ free(strand_names);
+ }
+ if (fai) {
+ fai_destroy(fai);
+ }
+ if (pool) {
+ hts_tpool_destroy(pool);
+ }
+ sam_global_args_free(&ga);
+ ks_free(&out.buffer);
return exit_status;
}
int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; }
const char* LZ4_versionString(void) { return LZ4_VERSION_STRING; }
int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); }
-int LZ4_sizeofState() { return LZ4_STREAMSIZE; }
+int LZ4_sizeofState(void) { return LZ4_STREAMSIZE; }
/*-******************************
/* Obsolete Streaming functions */
-int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; }
+int LZ4_sizeofStreamState(void) { return LZ4_STREAMSIZE; }
static void LZ4_init(LZ4_stream_t* lz4ds, BYTE* base)
{
int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; }
const char* LZ4_versionString(void) { return LZ4_VERSION_STRING; }
int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); }
-int LZ4_sizeofState() { return LZ4_STREAMSIZE; }
+int LZ4_sizeofState(void) { return LZ4_STREAMSIZE; }
/*-******************************
/* Obsolete Streaming functions */
-int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; }
+int LZ4_sizeofStreamState(void) { return LZ4_STREAMSIZE; }
static void LZ4_init(LZ4_stream_t* lz4ds, BYTE* base)
{
static int **count_all(int l, int vpos, nseq_t *hash)
{
khint_t k;
- int i, j, **cnt;
- uint8_t *seq;
+ int i, j, **cnt = NULL;
+ uint8_t *seq = NULL;
+ size_t cnt_sz = ((size_t)1) << l;
+ if (cnt_sz > SSIZE_MAX / sizeof(int) / vpos) {
+ errno = ENOMEM;
+ goto fail;
+ }
seq = calloc(l, 1);
+ if (!seq) goto fail;
cnt = calloc(vpos, sizeof(int*));
- for (i = 0; i < vpos; ++i) cnt[i] = calloc(1<<l, sizeof(int));
+ if (!cnt) goto fail;
+ for (i = 0; i < vpos; ++i) {
+ cnt[i] = calloc(cnt_sz, sizeof(int));
+ if (!cnt[i]) goto fail;
+ }
for (k = 0; k < kh_end(hash); ++k) {
if (kh_exist(hash, k)) {
frag_t *f = &kh_val(hash, k);
}
free(seq);
return cnt;
+ fail:
+ free(seq);
+ if (cnt) {
+ for (i = 0; i < vpos; i++)
+ free(cnt[i]);
+ free(cnt);
+ }
+ print_error_errno("phase", "Couldn't allocate memory for counts");
+ return NULL;
}
// phasing
printf("PS\t%s\t%d\t%d\n", chr, (int)(cns[0]>>32) + 1, (int)(cns[vpos-1]>>32) + 1);
sitemask = calloc(vpos, 1);
cnt = count_all(g->k, vpos, hash);
+ if (!cnt) return -1;
path = dynaprog(g->k, vpos, cnt);
for (i = 0; i < vpos; ++i) free(cnt[i]);
free(cnt);
static int **count_all(int l, int vpos, nseq_t *hash)
{
khint_t k;
- int i, j, **cnt;
- uint8_t *seq;
+ int i, j, **cnt = NULL;
+ uint8_t *seq = NULL;
+ size_t cnt_sz = ((size_t)1) << l;
+ if (cnt_sz > SSIZE_MAX / sizeof(int) / vpos) {
+ errno = ENOMEM;
+ goto fail;
+ }
seq = calloc(l, 1);
+ if (!seq) goto fail;
cnt = calloc(vpos, sizeof(int*));
- for (i = 0; i < vpos; ++i) cnt[i] = calloc(1<<l, sizeof(int));
+ if (!cnt) goto fail;
+ for (i = 0; i < vpos; ++i) {
+ cnt[i] = calloc(cnt_sz, sizeof(int));
+ if (!cnt[i]) goto fail;
+ }
for (k = 0; k < kh_end(hash); ++k) {
if (kh_exist(hash, k)) {
frag_t *f = &kh_val(hash, k);
}
free(seq);
return cnt;
+ fail:
+ free(seq);
+ if (cnt) {
+ for (i = 0; i < vpos; i++)
+ free(cnt[i]);
+ free(cnt);
+ }
+ print_error_errno("phase", "Couldn't allocate memory for counts");
+ return NULL;
}
// phasing
fprintf(samtools_stdout, "PS\t%s\t%d\t%d\n", chr, (int)(cns[0]>>32) + 1, (int)(cns[vpos-1]>>32) + 1);
sitemask = calloc(vpos, 1);
cnt = count_all(g->k, vpos, hash);
+ if (!cnt) return -1;
path = dynaprog(g->k, vpos, cnt);
for (i = 0; i < vpos; ++i) free(cnt[i]);
free(cnt);
/* reset.c -- removes aligner updates and reference data from input sam /
bam / cram file and makes read data raw for new processing
- Copyright (C) 2022, 2023 Genome Research Ltd.
+ Copyright (C) 2022 - 2024 Genome Research Ltd.
Author: Vasudeva Sarma <vasudeva.sarma@sanger.ac.uk>
#define TAGNUM(X) (((X)[0] << 8) | (X)[1]) //to create key for aux tags, like type key in htslib
#define LONG_OPT(X) (128 + (X)) //to handle long and short options with same char
+#define RESET_KEEPDUPFLAG 1 //keep dup flag as such, as in initial implementation
+
typedef struct conf_data
{
int keepRGs; //RG line handling
int noPGentry; //PG line for reset op or not
+ int ctrlFlags; //control flags
auxhash_t aux_keep; //SET that holds the aux tags to be retained
auxhash_t aux_remove; //SET that holds the aux tags to be removed
char *pgid; //PG id onwards which to be removed
--reject-PG ID\n\
Removes PG line with ID matching to input and succeeding PG lines\n\
--no-RG To have RG lines or not\n\
- --no-PG To have PG entry or not for reset operation\n");
+ --no-PG To have PG entry or not for reset operation\n\
+ --dupflag\n\
+ Keeps the duplicate flag as it is\n");
sam_global_opt_help(fp, "--O--@--");
return;
}
-/// removeauxtags - remove aux tags in bam data which are not present in acceptable tag set
-/** @param bamdata - pointer to the bamdata from which needs the filtering
- * @param config - pointer to conf_data
+/// update_aux_conf - update the user given aux tag configuration with defaults
+/** @param config - pointer to conf_data
returns nothing
*/
-void removeauxtags(bam1_t *bamdata, conf_data *config)
+void update_aux_conf(conf_data *config)
{
- uint8_t *auxdata = NULL;
- const char *tag = NULL, rg[] = "RG";
+ const char rg[] = "RG";
+ const char *default_tags[] = {"AS", "CC", "CG", "CP", "H1", "H2", "HI", "H0", "IH",
+ "MC", "MD", "MQ", "NM", "SA", "TS"};
khint_t iter = 0;
- int ret = 0;
+ int ret = 0, i = 0;
- if (!bamdata || !config || (!config->aux_keep && !config->aux_remove && config->keepRGs))
+ if (!config)
return;
- //remove RG tags from bamdata if keepRG is false
- if (!config->keepRGs) {
- if (!config->aux_keep && !config->aux_remove) {
- //none of aux tag filter in use, create remove filter
- config->aux_remove = kh_init(aux_exists);
- }
-
- if (config->aux_keep) {
- //keep set in use, remove RG if present
+ if (!config->aux_keep && !config->aux_remove) {
+ //none of aux tag filter in use, create remove filter
+ config->aux_remove = kh_init(aux_exists);
+ }
+ if (config->aux_keep) {
+ //keep set in use, remove RG if present
+ if (!config->keepRGs) {
iter = kh_get(aux_exists, config->aux_keep, TAGNUM(rg));
if (iter != kh_end(config->aux_keep)) {
kh_del(aux_exists, config->aux_keep, iter);
}
}
- if (config->aux_remove) {
+ }
+ if (config->aux_remove) {
+ if (!config->keepRGs) {
//remove set in use, add RG if not present
iter = kh_get(aux_exists, config->aux_remove, TAGNUM(rg));
if (iter == kh_end(config->aux_remove)) {
kh_put(aux_exists, config->aux_remove, TAGNUM(rg), &ret);
}
}
+ //add the default tags if not present in remove set
+ //note, keep has priority and this may not be honoured
+ for (i = 0; i < sizeof(default_tags) / sizeof(default_tags[0]); ++i) {
+ iter = kh_get(aux_exists, config->aux_remove, TAGNUM(default_tags[i]));
+ if (iter == kh_end(config->aux_remove)) {
+ kh_put(aux_exists, config->aux_remove, TAGNUM(default_tags[i]), &ret);
+ }
+ }
}
+}
+
+/// removeauxtags - remove aux tags in bam data which are not present in acceptable tag set
+/** @param bamdata - pointer to the bamdata from which needs the filtering
+ * @param config - pointer to conf_data
+returns nothing
+*/
+void removeauxtags(bam1_t *bamdata, conf_data *config)
+{
+ uint8_t *auxdata = NULL;
+ const char *tag = NULL;
+ khint_t iter = 0;
+
+ if (!bamdata || !config)
+ return;
for (auxdata = bam_aux_first(bamdata); auxdata; ) {
tag = bam_aux_tag(auxdata);
if (bamdata->core.flag & BAM_FSECONDARY || bamdata->core.flag & BAM_FSUPPLEMENTARY) {
continue;
}
-
//update flags
uint16_t flags = bamdata->core.flag & ~BAM_FPROPER_PAIR; //reset pair info
flags |= BAM_FUNMAP; //mark as unmapped
flags |= BAM_FMUNMAP; //mark mate as unmapped, if it was a pair
}
flags &= ~BAM_FMREVERSE; //reset mate orientation
+ if (!(config->ctrlFlags & RESET_KEEPDUPFLAG)) {
+ flags &= ~BAM_FDUP; //reset dup flag from alignment
+ }
if (0 > ks_resize(&querydata, bamdata->core.l_qseq) ||
0 > ks_resize(&qualdata, bamdata->core.l_qseq)) {
//reject PG lines from input, default is to keep them (i.e. option not given); without optional filename, all PGs removed and those given in file are filtered when optional filename is given
{"reject-PG", required_argument, NULL, 'p'}, //reject entries from this PG onwards
{"no-PG", no_argument, NULL, 2}, //do not add PG entry for reset operation, default is to add it
+ {"dupflag", no_argument, NULL, 3}, //keep the dup flag as it is - as in initial reset implementation
{NULL, 0, NULL, 0}
};
samFile *infile = NULL, *outfile = NULL;
const char *inname = NULL, *outname = NULL;
int c = 0, ret = EXIT_FAILURE;
char outmode[4] = "w", *args = NULL;
- conf_data resetconf = {1, 0, NULL, NULL, NULL}; //keep RGs and PGs by default
-
+ conf_data resetconf = {1, 0, 0, NULL, NULL, NULL}; //keep RGs and PGs by default, ctrlflags = 0
//samtools reset -o outfile -x/--remove-tag ... --keep-tag ... --threads=n --output-fmt=fmt --no-RG --reject-PG pgid --no-PG [<infile>]
while ((c = getopt_long(argc, argv, "o:@:x:O:", lopts, NULL)) >= 0)
}
resetconf.noPGentry = 1;
break;
+ case 3: //keep dup flag as it is / no reset
+ resetconf.ctrlFlags |= RESET_KEEPDUPFLAG;
+ break;
case 'p': //--reject-PG=<id>
if (resetconf.pgid) {
usage(stderr); //already given!
}
}
break;
- case LONG_OPT('x'): //keep aux tags
+ case LONG_OPT('x'): //keep aux tags
if (parse_aux_list(&resetconf.aux_keep, optarg, "main_reset")) {
usage(stderr);
goto exit;
inname = "-";
}
+ //update aux tag configuration
+ update_aux_conf(&resetconf);
//set output file format based on name
sam_open_mode(outmode + 1, outname, NULL);
/* reset.c -- removes aligner updates and reference data from input sam /
bam / cram file and makes read data raw for new processing
- Copyright (C) 2022, 2023 Genome Research Ltd.
+ Copyright (C) 2022 - 2024 Genome Research Ltd.
Author: Vasudeva Sarma <vasudeva.sarma@sanger.ac.uk>
#define TAGNUM(X) (((X)[0] << 8) | (X)[1]) //to create key for aux tags, like type key in htslib
#define LONG_OPT(X) (128 + (X)) //to handle long and short options with same char
+#define RESET_KEEPDUPFLAG 1 //keep dup flag as such, as in initial implementation
+
typedef struct conf_data
{
int keepRGs; //RG line handling
int noPGentry; //PG line for reset op or not
+ int ctrlFlags; //control flags
auxhash_t aux_keep; //SET that holds the aux tags to be retained
auxhash_t aux_remove; //SET that holds the aux tags to be removed
char *pgid; //PG id onwards which to be removed
--reject-PG ID\n\
Removes PG line with ID matching to input and succeeding PG lines\n\
--no-RG To have RG lines or not\n\
- --no-PG To have PG entry or not for reset operation\n");
+ --no-PG To have PG entry or not for reset operation\n\
+ --dupflag\n\
+ Keeps the duplicate flag as it is\n");
sam_global_opt_help(fp, "--O--@--");
return;
}
-/// removeauxtags - remove aux tags in bam data which are not present in acceptable tag set
-/** @param bamdata - pointer to the bamdata from which needs the filtering
- * @param config - pointer to conf_data
+/// update_aux_conf - update the user given aux tag configuration with defaults
+/** @param config - pointer to conf_data
returns nothing
*/
-void removeauxtags(bam1_t *bamdata, conf_data *config)
+void update_aux_conf(conf_data *config)
{
- uint8_t *auxdata = NULL;
- const char *tag = NULL, rg[] = "RG";
+ const char rg[] = "RG";
+ const char *default_tags[] = {"AS", "CC", "CG", "CP", "H1", "H2", "HI", "H0", "IH",
+ "MC", "MD", "MQ", "NM", "SA", "TS"};
khint_t iter = 0;
- int ret = 0;
+ int ret = 0, i = 0;
- if (!bamdata || !config || (!config->aux_keep && !config->aux_remove && config->keepRGs))
+ if (!config)
return;
- //remove RG tags from bamdata if keepRG is false
- if (!config->keepRGs) {
- if (!config->aux_keep && !config->aux_remove) {
- //none of aux tag filter in use, create remove filter
- config->aux_remove = kh_init(aux_exists);
- }
-
- if (config->aux_keep) {
- //keep set in use, remove RG if present
+ if (!config->aux_keep && !config->aux_remove) {
+ //none of aux tag filter in use, create remove filter
+ config->aux_remove = kh_init(aux_exists);
+ }
+ if (config->aux_keep) {
+ //keep set in use, remove RG if present
+ if (!config->keepRGs) {
iter = kh_get(aux_exists, config->aux_keep, TAGNUM(rg));
if (iter != kh_end(config->aux_keep)) {
kh_del(aux_exists, config->aux_keep, iter);
}
}
- if (config->aux_remove) {
+ }
+ if (config->aux_remove) {
+ if (!config->keepRGs) {
//remove set in use, add RG if not present
iter = kh_get(aux_exists, config->aux_remove, TAGNUM(rg));
if (iter == kh_end(config->aux_remove)) {
kh_put(aux_exists, config->aux_remove, TAGNUM(rg), &ret);
}
}
+ //add the default tags if not present in remove set
+ //note, keep has priority and this may not be honoured
+ for (i = 0; i < sizeof(default_tags) / sizeof(default_tags[0]); ++i) {
+ iter = kh_get(aux_exists, config->aux_remove, TAGNUM(default_tags[i]));
+ if (iter == kh_end(config->aux_remove)) {
+ kh_put(aux_exists, config->aux_remove, TAGNUM(default_tags[i]), &ret);
+ }
+ }
}
+}
+
+/// removeauxtags - remove aux tags in bam data which are not present in acceptable tag set
+/** @param bamdata - pointer to the bamdata from which needs the filtering
+ * @param config - pointer to conf_data
+returns nothing
+*/
+void removeauxtags(bam1_t *bamdata, conf_data *config)
+{
+ uint8_t *auxdata = NULL;
+ const char *tag = NULL;
+ khint_t iter = 0;
+
+ if (!bamdata || !config)
+ return;
for (auxdata = bam_aux_first(bamdata); auxdata; ) {
tag = bam_aux_tag(auxdata);
if (bamdata->core.flag & BAM_FSECONDARY || bamdata->core.flag & BAM_FSUPPLEMENTARY) {
continue;
}
-
//update flags
uint16_t flags = bamdata->core.flag & ~BAM_FPROPER_PAIR; //reset pair info
flags |= BAM_FUNMAP; //mark as unmapped
flags |= BAM_FMUNMAP; //mark mate as unmapped, if it was a pair
}
flags &= ~BAM_FMREVERSE; //reset mate orientation
+ if (!(config->ctrlFlags & RESET_KEEPDUPFLAG)) {
+ flags &= ~BAM_FDUP; //reset dup flag from alignment
+ }
if (0 > ks_resize(&querydata, bamdata->core.l_qseq) ||
0 > ks_resize(&qualdata, bamdata->core.l_qseq)) {
//reject PG lines from input, default is to keep them (i.e. option not given); without optional filename, all PGs removed and those given in file are filtered when optional filename is given
{"reject-PG", required_argument, NULL, 'p'}, //reject entries from this PG onwards
{"no-PG", no_argument, NULL, 2}, //do not add PG entry for reset operation, default is to add it
+ {"dupflag", no_argument, NULL, 3}, //keep the dup flag as it is - as in initial reset implementation
{NULL, 0, NULL, 0}
};
samFile *infile = NULL, *outfile = NULL;
const char *inname = NULL, *outname = NULL;
int c = 0, ret = EXIT_FAILURE;
char outmode[4] = "w", *args = NULL;
- conf_data resetconf = {1, 0, NULL, NULL, NULL}; //keep RGs and PGs by default
-
+ conf_data resetconf = {1, 0, 0, NULL, NULL, NULL}; //keep RGs and PGs by default, ctrlflags = 0
//samtools reset -o outfile -x/--remove-tag ... --keep-tag ... --threads=n --output-fmt=fmt --no-RG --reject-PG pgid --no-PG [<infile>]
while ((c = getopt_long(argc, argv, "o:@:x:O:", lopts, NULL)) >= 0)
}
resetconf.noPGentry = 1;
break;
+ case 3: //keep dup flag as it is / no reset
+ resetconf.ctrlFlags |= RESET_KEEPDUPFLAG;
+ break;
case 'p': //--reject-PG=<id>
if (resetconf.pgid) {
usage(samtools_stderr); //already given!
}
}
break;
- case LONG_OPT('x'): //keep aux tags
+ case LONG_OPT('x'): //keep aux tags
if (parse_aux_list(&resetconf.aux_keep, optarg, "main_reset")) {
usage(samtools_stderr);
goto exit;
inname = "-";
}
+ //update aux tag configuration
+ update_aux_conf(&resetconf);
//set output file format based on name
sam_open_mode(outmode + 1, outname, NULL);
if (samtools_stdout == fp) samtools_stdout = NULL;
}
-static void vprint_error_core(const char *subcommand, const char *format, va_list args, const char *extra)
+static void HTS_FORMAT(HTS_PRINTF_FMT, 2, 0)
+vprint_error_core(const char *subcommand, const char *format, va_list args, const char *extra)
{
fflush(stdout);
if (samtools_stdout) hts_flush(samtools_stdout);
if (samtools_stdout_internal == fp) samtools_stdout_internal = NULL;
}
-static void vprint_error_core(const char *subcommand, const char *format, va_list args, const char *extra)
+static void HTS_FORMAT(HTS_PRINTF_FMT, 2, 0)
+vprint_error_core(const char *subcommand, const char *format, va_list args, const char *extra)
{
fflush(samtools_stdout);
if (samtools_stdout_internal) hts_flush(samtools_stdout_internal);
/* sam_view.c -- SAM<->BAM<->CRAM conversion.
- Copyright (C) 2009-2023 Genome Research Ltd.
+ Copyright (C) 2009-2024 Genome Research Ltd.
Portions copyright (C) 2009, 2011, 2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
strhash_t rnhash;
strhash_t tvhash;
int min_mapQ;
+ int rghash_discard; // 0 keep, 1 discard
+ int rnhash_discard; // 0 keep, 1 discard
// Described here in the same terms as the usage statement.
// The code however always negates to "reject if" keep if:
uint8_t *s = bam_aux_get(b, "RG");
if (s) {
khint_t k = kh_get(str, settings->rghash, (char*)(s + 1));
- if (k == kh_end(settings->rghash)) return 1;
+ if ((k == kh_end(settings->rghash)) != settings->rghash_discard)
+ return 1;
}
}
if (settings->tag) {
}
if (settings->rnhash) {
const char* rn = bam_get_qname(b);
- if (!rn || kh_get(str, settings->rnhash, rn) == kh_end(settings->rnhash)) {
+ strhash_t h = settings->rnhash;
+ if (!rn && !settings->rnhash_discard)
+ return 1;
+ if ((kh_get(str, h, rn) == kh_end(h)) != settings->rnhash_discard)
return 1;
- }
}
if (settings->library) {
const char *p = bam_get_library((sam_hdr_t*)h, b);
if (settings->rghash == NULL) {
settings->rghash = kh_init(str);
if (settings->rghash == NULL) goto err;
+ } else if (settings->rghash_discard == 1) {
+ print_error("view", "cannot mix include and exclude read-group files in the same command line");
+ free(d);
+ return -1;
}
+ settings->rghash_discard = 0;
kh_put(str, settings->rghash, d, &ret);
if (ret == -1) goto err;
perror(NULL);
return -1;
}
+ } else if ((settings->rnhash_discard == 0 && *fn == '^') ||
+ (settings->rnhash_discard == 1 && *fn != '^')) {
+ print_error("view", "cannot mix include and exclude read-name files in the same command line");
+ return -1;
}
- return populate_lookup_from_file(subcmd, settings->rnhash, fn);
+ settings->rnhash_discard = (*fn == '^');
+ return populate_lookup_from_file(subcmd, settings->rnhash,
+ fn + (*fn == '^'));
}
static int add_read_groups_file(const char *subcmd, samview_settings_t *settings, char *fn)
perror(NULL);
return -1;
}
+ } else if ((settings->rghash_discard == 0 && *fn == '^') ||
+ (settings->rghash_discard == 1 && *fn != '^')) {
+ print_error("view", "cannot mix include and exclude read-group files in the same command line");
+ return -1;
}
- return populate_lookup_from_file(subcmd, settings->rghash, fn);
+ settings->rghash_discard = (*fn == '^');
+ return populate_lookup_from_file(subcmd, settings->rghash,
+ fn + (*fn == '^'));
}
static int add_tag_value_single(const char *subcmd, samview_settings_t *settings, char *name)
int filter_state = ALL;
if ( nregs ) {
int filter_op = 0;
- conf->bed = bed_hash_regions(conf->bed, regs, 0, nregs, &filter_op); // insert(1) or filter out(0) the regions from the command line in the same hash table as the bed file
+ void *bed = bed_hash_regions(conf->bed, regs, 0, nregs, &filter_op); // insert(1) or filter out(0) the regions from the command line in the same hash table as the bed file
+ if (!bed) {
+ print_error_errno("view", "Couldn't %s region list",
+ filter_op ? "build" : "filter");
+ return NULL;
+ }
+ conf->bed = bed;
if ( !filter_op )
filter_state = FILTERED;
}
else
bed_unify(conf->bed);
+
+ // This check is probably redundant, but left just in case
if ( !conf->bed) { // index is unavailable or no regions have been specified
print_error("view", "No regions or BED file have been provided. Aborting.");
return NULL;
}
}
- if ( rec->core.mtid < 0 || (rec->core.flag & BAM_FMUNMAP) ) nunmap = 1;
- if ( rec->core.mtid >= 0 ) {
+ if ( rec->core.mtid < 0 ) {
+ nunmap = 1;
+ } else {
if (_reglist_push(&conf->reglist, &conf->nreglist, rec->core.mtid, rec->core.mpos,rec->core.mpos+1) != 0)
goto out;
}
while ((result = sam_itr_multi_next(conf->in, iter, b)) >= 0) {
if (process_one_record(conf, b, &write_error) < 0) break;
}
- hts_itr_multi_destroy(iter);
bam_destroy1(b);
if (result < -1) {
print_error("view", "retrieval of region #%d failed", iter->curr_tid);
- return 1;
+ write_error = 1;
}
+ hts_itr_multi_destroy(iter);
return write_error;
}
{"use-index", no_argument, NULL, 'M'},
{"with-header", no_argument, NULL, 'h'},
{"sanitize", required_argument, NULL, 'z'},
+ {NULL, 0, NULL, 0}
};
/* parse command-line options */
opterr = 0;
char *tmp;
+ int tmp_flag;
+
while ((c = getopt_long(argc, argv,
"SbBcCt:h1Ho:O:q:f:F:G:ul:r:T:R:N:d:D:L:s:@:m:x:U:MXe:pPz:",
lopts, NULL)) >= 0) {
case 'U': settings.fn_un_out = strdup(optarg); break;
case 'X': has_index_file = 1; break;
case 'f':
- settings.flag_on |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_on |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case 'F':
- settings.flag_off |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_off |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case LONGOPT('g'):
- settings.flag_anyon |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_anyon |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case 'G':
- settings.flag_alloff |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_alloff |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case 'q':
}
settings.count_rf = INT_MAX; // no way to know what we need
break;
- case LONGOPT('r'): settings.remove_flag |= bam_str2flag(optarg); break;
- case LONGOPT('a'): settings.add_flag |= bam_str2flag(optarg); break;
+ case LONGOPT('r'):
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.remove_flag |= tmp_flag;
+ break;
+
+ case LONGOPT('a'):
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.add_flag |= tmp_flag;
+ break;
case 'x':
if (*optarg == '^') {
}
}
- if ( settings.hts_idx ) hts_idx_destroy(settings.hts_idx);
-
if (ga.write_index) {
if (sam_idx_save(settings.out) < 0) {
print_error_errno("view", "writing index failed");
}
view_end:
+ if ( settings.hts_idx ) hts_idx_destroy(settings.hts_idx);
+
if (settings.is_count && ret == 0) {
if (fprintf(settings.fn_out? fp_out : stdout, "%" PRId64 "\n", settings.count) < 0) {
if (settings.fn_out) print_error_errno("view", "writing to \"%s\" failed", settings.fn_out);
"\n"
"Filtering options (Only include in output reads that...):\n"
" -L, --target[s]-file FILE ...overlap (BED) regions in FILE\n"
+" -N, --qname-file [^]FILE ...whose read name is listed in FILE (\"^\" negates)\n"
" -r, --read-group STR ...are in read group STR\n"
-" -R, --read-group-file FILE ...are in a read group listed in FILE\n"
-" -N, --qname-file FILE ...whose read name is listed in FILE\n"
+" -R, --read-group-file [^]FILE\n"
+" ...are in a read group listed in FILE\n"
" -d, --tag STR1[:STR2] ...have a tag STR1 (with associated value STR2)\n"
" -D, --tag-file STR:FILE ...have a tag STR whose value is listed in FILE\n"
" -q, --min-MQ INT ...have mapping quality >= INT\n"
if (nrecords > 0) {
b = bam_init1();
uint64_t n;
- int r;
+ int r = 0;
for (n = 0; n < nrecords && (r = sam_read1(fp, hdr, b)) >= 0; n++) {
if (sam_format1(hdr, b, &str) < 0) {
print_error_errno("head", "couldn't format record");
/* sam_view.c -- SAM<->BAM<->CRAM conversion.
- Copyright (C) 2009-2023 Genome Research Ltd.
+ Copyright (C) 2009-2024 Genome Research Ltd.
Portions copyright (C) 2009, 2011, 2012 Broad Institute.
Author: Heng Li <lh3@sanger.ac.uk>
strhash_t rnhash;
strhash_t tvhash;
int min_mapQ;
+ int rghash_discard; // 0 keep, 1 discard
+ int rnhash_discard; // 0 keep, 1 discard
// Described here in the same terms as the usage statement.
// The code however always negates to "reject if" keep if:
uint8_t *s = bam_aux_get(b, "RG");
if (s) {
khint_t k = kh_get(str, settings->rghash, (char*)(s + 1));
- if (k == kh_end(settings->rghash)) return 1;
+ if ((k == kh_end(settings->rghash)) != settings->rghash_discard)
+ return 1;
}
}
if (settings->tag) {
}
if (settings->rnhash) {
const char* rn = bam_get_qname(b);
- if (!rn || kh_get(str, settings->rnhash, rn) == kh_end(settings->rnhash)) {
+ strhash_t h = settings->rnhash;
+ if (!rn && !settings->rnhash_discard)
+ return 1;
+ if ((kh_get(str, h, rn) == kh_end(h)) != settings->rnhash_discard)
return 1;
- }
}
if (settings->library) {
const char *p = bam_get_library((sam_hdr_t*)h, b);
if (settings->rghash == NULL) {
settings->rghash = kh_init(str);
if (settings->rghash == NULL) goto err;
+ } else if (settings->rghash_discard == 1) {
+ print_error("view", "cannot mix include and exclude read-group files in the same command line");
+ free(d);
+ return -1;
}
+ settings->rghash_discard = 0;
kh_put(str, settings->rghash, d, &ret);
if (ret == -1) goto err;
perror(NULL);
return -1;
}
+ } else if ((settings->rnhash_discard == 0 && *fn == '^') ||
+ (settings->rnhash_discard == 1 && *fn != '^')) {
+ print_error("view", "cannot mix include and exclude read-name files in the same command line");
+ return -1;
}
- return populate_lookup_from_file(subcmd, settings->rnhash, fn);
+ settings->rnhash_discard = (*fn == '^');
+ return populate_lookup_from_file(subcmd, settings->rnhash,
+ fn + (*fn == '^'));
}
static int add_read_groups_file(const char *subcmd, samview_settings_t *settings, char *fn)
perror(NULL);
return -1;
}
+ } else if ((settings->rghash_discard == 0 && *fn == '^') ||
+ (settings->rghash_discard == 1 && *fn != '^')) {
+ print_error("view", "cannot mix include and exclude read-group files in the same command line");
+ return -1;
}
- return populate_lookup_from_file(subcmd, settings->rghash, fn);
+ settings->rghash_discard = (*fn == '^');
+ return populate_lookup_from_file(subcmd, settings->rghash,
+ fn + (*fn == '^'));
}
static int add_tag_value_single(const char *subcmd, samview_settings_t *settings, char *name)
int filter_state = ALL;
if ( nregs ) {
int filter_op = 0;
- conf->bed = bed_hash_regions(conf->bed, regs, 0, nregs, &filter_op); // insert(1) or filter out(0) the regions from the command line in the same hash table as the bed file
+ void *bed = bed_hash_regions(conf->bed, regs, 0, nregs, &filter_op); // insert(1) or filter out(0) the regions from the command line in the same hash table as the bed file
+ if (!bed) {
+ print_error_errno("view", "Couldn't %s region list",
+ filter_op ? "build" : "filter");
+ return NULL;
+ }
+ conf->bed = bed;
if ( !filter_op )
filter_state = FILTERED;
}
else
bed_unify(conf->bed);
+
+ // This check is probably redundant, but left just in case
if ( !conf->bed) { // index is unavailable or no regions have been specified
print_error("view", "No regions or BED file have been provided. Aborting.");
return NULL;
}
}
- if ( rec->core.mtid < 0 || (rec->core.flag & BAM_FMUNMAP) ) nunmap = 1;
- if ( rec->core.mtid >= 0 ) {
+ if ( rec->core.mtid < 0 ) {
+ nunmap = 1;
+ } else {
if (_reglist_push(&conf->reglist, &conf->nreglist, rec->core.mtid, rec->core.mpos,rec->core.mpos+1) != 0)
goto out;
}
while ((result = sam_itr_multi_next(conf->in, iter, b)) >= 0) {
if (process_one_record(conf, b, &write_error) < 0) break;
}
- hts_itr_multi_destroy(iter);
bam_destroy1(b);
if (result < -1) {
print_error("view", "retrieval of region #%d failed", iter->curr_tid);
- return 1;
+ write_error = 1;
}
+ hts_itr_multi_destroy(iter);
return write_error;
}
{"use-index", no_argument, NULL, 'M'},
{"with-header", no_argument, NULL, 'h'},
{"sanitize", required_argument, NULL, 'z'},
+ {NULL, 0, NULL, 0}
};
/* parse command-line options */
opterr = 0;
char *tmp;
+ int tmp_flag;
+
while ((c = getopt_long(argc, argv,
"SbBcCt:h1Ho:O:q:f:F:G:ul:r:T:R:N:d:D:L:s:@:m:x:U:MXe:pPz:",
lopts, NULL)) >= 0) {
case 'U': settings.fn_un_out = strdup(optarg); break;
case 'X': has_index_file = 1; break;
case 'f':
- settings.flag_on |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_on |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case 'F':
- settings.flag_off |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_off |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case LONGOPT('g'):
- settings.flag_anyon |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_anyon |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case 'G':
- settings.flag_alloff |= bam_str2flag(optarg);
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.flag_alloff |= tmp_flag;
settings.count_rf |= SAM_FLAG | SAM_RNEXT;
break;
case 'q':
}
settings.count_rf = INT_MAX; // no way to know what we need
break;
- case LONGOPT('r'): settings.remove_flag |= bam_str2flag(optarg); break;
- case LONGOPT('a'): settings.add_flag |= bam_str2flag(optarg); break;
+ case LONGOPT('r'):
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.remove_flag |= tmp_flag;
+ break;
+
+ case LONGOPT('a'):
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("view", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ settings.add_flag |= tmp_flag;
+ break;
case 'x':
if (*optarg == '^') {
}
}
- if ( settings.hts_idx ) hts_idx_destroy(settings.hts_idx);
-
if (ga.write_index) {
if (sam_idx_save(settings.out) < 0) {
print_error_errno("view", "writing index failed");
}
view_end:
+ if ( settings.hts_idx ) hts_idx_destroy(settings.hts_idx);
+
if (settings.is_count && ret == 0) {
if (fprintf(settings.fn_out? fp_out : samtools_stdout, "%" PRId64 "\n", settings.count) < 0) {
if (settings.fn_out) print_error_errno("view", "writing to \"%s\" failed", settings.fn_out);
"\n"
"Filtering options (Only include in output reads that...):\n"
" -L, --target[s]-file FILE ...overlap (BED) regions in FILE\n"
+" -N, --qname-file [^]FILE ...whose read name is listed in FILE (\"^\" negates)\n"
" -r, --read-group STR ...are in read group STR\n"
-" -R, --read-group-file FILE ...are in a read group listed in FILE\n"
-" -N, --qname-file FILE ...whose read name is listed in FILE\n"
+" -R, --read-group-file [^]FILE\n"
+" ...are in a read group listed in FILE\n"
" -d, --tag STR1[:STR2] ...have a tag STR1 (with associated value STR2)\n"
" -D, --tag-file STR:FILE ...have a tag STR whose value is listed in FILE\n"
" -q, --min-MQ INT ...have mapping quality >= INT\n"
if (nrecords > 0) {
b = bam_init1();
uint64_t n;
- int r;
+ int r = 0;
for (n = 0; n < nrecords && (r = sam_read1(fp, hdr, b)) >= 0; n++) {
if (sam_format1(hdr, b, &str) < 0) {
print_error_errno("head", "couldn't format record");
/* stats.c -- This is the former bamcheck integrated into samtools/htslib.
- Copyright (C) 2012-2022 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
Author: Sam Nicholls <sam@samnicholls.net>
continue;
uint32_t barcode_len = strlen(barcode);
+ if (!barcode_len) {
+ continue; //consider 0 size barcode same as no barcode - avoids issues with realloc below
+ }
if (!stats->tags_barcode[tag].nbases) { // tag seen for the first time
uint32_t offset = 0;
for (i = 0; i < stats->ntags; i++)
}
-static void HTS_NORETURN error(const char *format, ...)
+static void HTS_FORMAT(HTS_PRINTF_FMT, 1, 2) HTS_NORETURN
+error(const char *format, ...)
{
if ( !format )
{
void cleanup_stats_info(stats_info_t* info){
if (info->fai) fai_destroy(info->fai);
- sam_close(info->sam);
+ if (info->sam) sam_close(info->sam);
free(info);
}
return 0;
}
-stats_t* stats_init()
+stats_t* stats_init(void)
{
stats_t *stats = calloc(1,sizeof(stats_t));
if (!stats)
{"cov-threshold", required_argument, NULL, 'g'},
{NULL, 0, NULL, 0}
};
- int opt;
+ int opt, tmp_flag;
while ( (opt=getopt_long(argc,argv,"?hdsXxpr:c:l:i:t:m:q:f:F:g:I:S:P:@:",loptions,NULL))>0 )
{
switch (opt)
{
- case 'f': info->flag_require = bam_str2flag(optarg); break;
- case 'F': info->flag_filter |= bam_str2flag(optarg); break;
+ case 'f':
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("stats", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ info->flag_require = tmp_flag;
+ break;
+
+ case 'F':
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("stats", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ info->flag_filter |= tmp_flag;
+ break;
+
case 'd': info->flag_filter |= BAM_FDUP; break;
case 'X': has_index_file = 1; break;
case 's': break;
/* stats.c -- This is the former bamcheck integrated into samtools/htslib.
- Copyright (C) 2012-2022 Genome Research Ltd.
+ Copyright (C) 2012-2024 Genome Research Ltd.
Author: Petr Danecek <pd3@sanger.ac.uk>
Author: Sam Nicholls <sam@samnicholls.net>
continue;
uint32_t barcode_len = strlen(barcode);
+ if (!barcode_len) {
+ continue; //consider 0 size barcode same as no barcode - avoids issues with realloc below
+ }
if (!stats->tags_barcode[tag].nbases) { // tag seen for the first time
uint32_t offset = 0;
for (i = 0; i < stats->ntags; i++)
}
-static void HTS_NORETURN error(const char *format, ...)
+static void HTS_FORMAT(HTS_PRINTF_FMT, 1, 2) HTS_NORETURN
+error(const char *format, ...)
{
if ( !format )
{
void cleanup_stats_info(stats_info_t* info){
if (info->fai) fai_destroy(info->fai);
- sam_close(info->sam);
+ if (info->sam) sam_close(info->sam);
free(info);
}
return 0;
}
-stats_t* stats_init()
+stats_t* stats_init(void)
{
stats_t *stats = calloc(1,sizeof(stats_t));
if (!stats)
{"cov-threshold", required_argument, NULL, 'g'},
{NULL, 0, NULL, 0}
};
- int opt;
+ int opt, tmp_flag;
while ( (opt=getopt_long(argc,argv,"?hdsXxpr:c:l:i:t:m:q:f:F:g:I:S:P:@:",loptions,NULL))>0 )
{
switch (opt)
{
- case 'f': info->flag_require = bam_str2flag(optarg); break;
- case 'F': info->flag_filter |= bam_str2flag(optarg); break;
+ case 'f':
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("stats", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ info->flag_require = tmp_flag;
+ break;
+
+ case 'F':
+ tmp_flag = bam_str2flag(optarg);
+
+ if (tmp_flag < 0) {
+ print_error("stats", "Unknown flag '%s'", optarg);
+ return 1;
+ }
+
+ info->flag_filter |= tmp_flag;
+ break;
+
case 'd': info->flag_filter |= BAM_FDUP; break;
case 'X': has_index_file = 1; break;
case 's': break;
#include "htslib/sam.h"
-static void tmp_print_error(tmp_file_t *tmp, const char *fmt, ...) {
+static void HTS_FORMAT(HTS_PRINTF_FMT, 2, 3)
+tmp_print_error(tmp_file_t *tmp, const char *fmt, ...) {
va_list argp;
if (tmp->verbose) {
int ret;
if ((ret = tmp_file_grow_ring_buffer(tmp, (tmp->offset + tmp->input_size + sizeof(bam1_t) + inbam->l_data) * 2))) {
- tmp_print_error(tmp, "[tmp_file] Error: input line too big. (%ld).\n",
- (tmp->input_size + inbam->l_data));
+ tmp_print_error(tmp, "[tmp_file] Error: input line too big. (%zu).\n",
+ (tmp->input_size + inbam->l_data));
return ret;
}
tmp->entry_number++;
if (tmp->read_size > tmp->output_size) {
- tmp_print_error(tmp, "[tmp_file] Error: wrong size of data returned RS:%ld OS:%ld EN:%ld GS:%ld.\n",
+ tmp_print_error(tmp, "[tmp_file] Error: wrong size of data returned RS:%zu OS:%zu EN:%zu GS:%zu.\n",
tmp->read_size, tmp->output_size, tmp->entry_number, tmp->group_size);
return TMP_SAM_LZ4_ERROR;
}
#include "htslib/sam.h"
-static void tmp_print_error(tmp_file_t *tmp, const char *fmt, ...) {
+static void HTS_FORMAT(HTS_PRINTF_FMT, 2, 3)
+tmp_print_error(tmp_file_t *tmp, const char *fmt, ...) {
va_list argp;
if (tmp->verbose) {
int ret;
if ((ret = tmp_file_grow_ring_buffer(tmp, (tmp->offset + tmp->input_size + sizeof(bam1_t) + inbam->l_data) * 2))) {
- tmp_print_error(tmp, "[tmp_file] Error: input line too big. (%ld).\n",
- (tmp->input_size + inbam->l_data));
+ tmp_print_error(tmp, "[tmp_file] Error: input line too big. (%zu).\n",
+ (tmp->input_size + inbam->l_data));
return ret;
}
tmp->entry_number++;
if (tmp->read_size > tmp->output_size) {
- tmp_print_error(tmp, "[tmp_file] Error: wrong size of data returned RS:%ld OS:%ld EN:%ld GS:%ld.\n",
+ tmp_print_error(tmp, "[tmp_file] Error: wrong size of data returned RS:%zu OS:%zu EN:%zu GS:%zu.\n",
tmp->read_size, tmp->output_size, tmp->entry_number, tmp->group_size);
return TMP_SAM_LZ4_ERROR;
}
# DEALINGS IN THE SOFTWARE.
# Master version, for use in tarballs or non-git source copies
-VERSION=1.18
+VERSION=1.21
# If we have a git clone, then check against the current tag
if [ -e .git ]
del os.environ[var]
+def format_macro_option(name, value):
+ return f"-D{name}={value}" if value is not None else f"-D{name}"
+
+
def configure_library(library_dir, env_options=None, options=[]):
configure_script = os.path.join(library_dir, "configure")
define_macros = []
+if os.environ.get("CIBUILDWHEEL", "0") == "1":
+ define_macros.append(("BUILDING_WHEEL", None))
+
suffix = sysconfig.get_config_var('EXT_SUFFIX')
internal_htslib_libraries = [
# TODO Eventually by running configure here, we can set these
# extra flags for configure instead of hacking on ALL_CPPFLAGS.
args = " ".join(ext.extra_compile_args)
- run_make(["ALL_CPPFLAGS=-I. " + args + " $(CPPFLAGS)", "lib-static"])
+ defines = " ".join([format_macro_option(*pair) for pair in ext.define_macros])
+ run_make(["ALL_CPPFLAGS=-I. " + args + " " + defines + " $(CPPFLAGS)", "lib-static"])
else:
log.warning("skipping 'libhts.a' (already built)")
import struct
import copy
import array
+from pysam import CDEL, CDIFF, CEQUAL, CINS, CMATCH, CPAD, CREF_SKIP, CSOFT_CLIP
from TestUtils import (
checkFieldEqual,
and manipulated.
"""
+ def check_get_aligned_pairs_combos(self, a, exp):
+ def positions(exp): return [(ppos, rpos) for ppos, rpos, base, cigar in exp]
+ def with_seq(exp): return [(ppos, rpos, base) for ppos, rpos, base, cigar in exp]
+ def with_cigar(exp): return [(ppos, rpos, cigar) for ppos, rpos, base, cigar in exp]
+
+ self.assertEqual(a.get_aligned_pairs(), positions(exp))
+ self.assertEqual(a.get_aligned_pairs(with_seq=True), with_seq(exp))
+ self.assertEqual(a.get_aligned_pairs(with_cigar=True), with_cigar(exp))
+ self.assertEqual(a.get_aligned_pairs(with_seq=True, with_cigar=True), exp)
+
+ exp = [(ppos, rpos, base, cigar) for ppos, rpos, base, cigar in exp if ppos is not None and rpos is not None]
+
+ self.assertEqual(a.get_aligned_pairs(matches_only=True), positions(exp))
+ self.assertEqual(a.get_aligned_pairs(matches_only=True, with_seq=True), with_seq(exp))
+ self.assertEqual(a.get_aligned_pairs(matches_only=True, with_cigar=True), with_cigar(exp))
+ self.assertEqual(a.get_aligned_pairs(matches_only=True, with_seq=True, with_cigar=True), exp)
+
+ def check_get_aligned_pairs_combos_without_MD(self, a, exp):
+ def positions(exp): return [(ppos, rpos) for ppos, rpos, cigar in exp]
+
+ self.assertEqual(a.get_aligned_pairs(), positions(exp))
+ with self.assertRaises(ValueError): a.get_aligned_pairs(with_seq=True)
+ self.assertEqual(a.get_aligned_pairs(with_cigar=True), exp)
+ with self.assertRaises(ValueError): a.get_aligned_pairs(with_seq=True, with_cigar=True)
+
+ exp = [(ppos, rpos, cigar) for ppos, rpos, cigar in exp if ppos is not None and rpos is not None]
+
+ self.assertEqual(a.get_aligned_pairs(matches_only=True), positions(exp))
+ with self.assertRaises(ValueError): a.get_aligned_pairs(matches_only=True, with_seq=True)
+ self.assertEqual(a.get_aligned_pairs(matches_only=True, with_cigar=True), exp)
+ with self.assertRaises(ValueError): a.get_aligned_pairs(matches_only=True, with_seq=True, with_cigar=True)
+
def testEmpty(self):
a = pysam.AlignedSegment()
],
)
- self.assertEqual(
- a.get_aligned_pairs(),
+ self.check_get_aligned_pairs_combos_without_MD(
+ a,
[
- (0, 20),
- (1, 21),
- (2, 22),
- (3, 23),
- (4, 24),
- (5, 25),
- (6, 26),
- (7, 27),
- (8, 28),
- (9, 29),
- (None, 30),
- (10, 31),
- (11, 32),
- (12, 33),
- (13, 34),
- (14, 35),
- (15, 36),
- (16, 37),
- (17, 38),
- (18, 39),
- (19, None),
- (20, 40),
- (21, 41),
- (22, 42),
- (23, 43),
- (24, 44),
- (25, 45),
- (26, 46),
- (27, 47),
- (28, 48),
- (29, 49),
- (30, 50),
- (31, 51),
- (32, 52),
- (33, 53),
- (34, 54),
- (35, 55),
- (36, 56),
- (37, 57),
- (38, 58),
- (39, 59),
+ (0, 20, CMATCH),
+ (1, 21, CMATCH),
+ (2, 22, CMATCH),
+ (3, 23, CMATCH),
+ (4, 24, CMATCH),
+ (5, 25, CMATCH),
+ (6, 26, CMATCH),
+ (7, 27, CMATCH),
+ (8, 28, CMATCH),
+ (9, 29, CMATCH),
+ (None, 30, CDEL),
+ (10, 31, CMATCH),
+ (11, 32, CMATCH),
+ (12, 33, CMATCH),
+ (13, 34, CMATCH),
+ (14, 35, CMATCH),
+ (15, 36, CMATCH),
+ (16, 37, CMATCH),
+ (17, 38, CMATCH),
+ (18, 39, CMATCH),
+ (19, None, CINS),
+ (20, 40, CMATCH),
+ (21, 41, CMATCH),
+ (22, 42, CMATCH),
+ (23, 43, CMATCH),
+ (24, 44, CMATCH),
+ (25, 45, CMATCH),
+ (26, 46, CMATCH),
+ (27, 47, CMATCH),
+ (28, 48, CMATCH),
+ (29, 49, CMATCH),
+ (30, 50, CMATCH),
+ (31, 51, CMATCH),
+ (32, 52, CMATCH),
+ (33, 53, CMATCH),
+ (34, 54, CMATCH),
+ (35, 55, CMATCH),
+ (36, 56, CMATCH),
+ (37, 57, CMATCH),
+ (38, 58, CMATCH),
+ (39, 59, CMATCH),
],
)
def test_get_aligned_pairs_soft_clipping(self):
a = self.build_read()
a.cigartuples = ((4, 2), (0, 35), (4, 3))
- self.assertEqual(
- a.get_aligned_pairs(),
- [(0, None), (1, None)]
+ self.check_get_aligned_pairs_combos_without_MD(
+ a,
+ [(0, None, CSOFT_CLIP), (1, None, CSOFT_CLIP)]
+ [
- (qpos, refpos)
+ (qpos, refpos, CMATCH)
for (qpos, refpos) in zip(range(2, 2 + 35), range(20, 20 + 35))
]
- + [(37, None), (38, None), (39, None)],
- )
- self.assertEqual(
- a.get_aligned_pairs(True),
- # [(0, None), (1, None)] +
- [
- (qpos, refpos)
- for (qpos, refpos) in zip(range(2, 2 + 35), range(20, 20 + 35))
- ]
- # [(37, None), (38, None), (39, None)]
+ + [(37, None, CSOFT_CLIP), (38, None, CSOFT_CLIP), (39, None, CSOFT_CLIP)],
)
def test_get_aligned_pairs_hard_clipping(self):
a = self.build_read()
a.cigartuples = ((5, 2), (0, 35), (5, 3))
- self.assertEqual(
- a.get_aligned_pairs(),
+ self.check_get_aligned_pairs_combos_without_MD(
+ a,
# No seq, no seq pos
[
- (qpos, refpos)
- for (qpos, refpos) in zip(range(0, 0 + 35), range(20, 20 + 35))
- ],
- )
- self.assertEqual(
- a.get_aligned_pairs(True),
- [
- (qpos, refpos)
+ (qpos, refpos, CMATCH)
for (qpos, refpos) in zip(range(0, 0 + 35), range(20, 20 + 35))
],
)
def test_get_aligned_pairs_skip(self):
a = self.build_read()
a.cigarstring = "2M100D38M"
- self.assertEqual(
- a.get_aligned_pairs(),
- [(0, 20), (1, 21)]
- + [(None, refpos) for refpos in range(22, 22 + 100)]
+ self.check_get_aligned_pairs_combos_without_MD(
+ a,
+ [(0, 20, CMATCH), (1, 21, CMATCH)]
+ + [(None, refpos, CDEL) for refpos in range(22, 22 + 100)]
+ [
- (qpos, refpos)
- for (qpos, refpos) in zip(
- range(2, 2 + 38), range(20 + 2 + 100, 20 + 2 + 100 + 38)
- )
- ],
- )
- self.assertEqual(
- a.get_aligned_pairs(True),
- [(0, 20), (1, 21)] +
- # [(None, refpos) for refpos in range(21, 21+100)] +
- [
- (qpos, refpos)
+ (qpos, refpos, CMATCH)
for (qpos, refpos) in zip(
range(2, 2 + 38), range(20 + 2 + 100, 20 + 2 + 100 + 38)
)
def test_get_aligned_pairs_match_mismatch(self):
a = self.build_read()
a.cigartuples = ((7, 20), (8, 20))
- self.assertEqual(
- a.get_aligned_pairs(),
- [
- (qpos, refpos)
- for (qpos, refpos) in zip(range(0, 0 + 40), range(20, 20 + 40))
- ],
- )
- self.assertEqual(
- a.get_aligned_pairs(True),
+ self.check_get_aligned_pairs_combos_without_MD(
+ a,
[
- (qpos, refpos)
+ (qpos, refpos, CEQUAL if qpos < 20 else CDIFF)
for (qpos, refpos) in zip(range(0, 0 + 40), range(20, 20 + 40))
],
)
a.cigartuples = ((0, 1), (6, 1), (0, 1))
# The padding operation is like an insertion into the reference.
# See comment in test_get_aligned_pairs_padding_with_seq (below).
- self.assertEqual(a.get_aligned_pairs(),
- [(0, 20), (1, None), (2, 21)])
+ self.check_get_aligned_pairs_combos_without_MD(a,
+ [(0, 20, CMATCH), (1, None, CPAD), (2, 21, CMATCH)])
+
+ def test_get_aligned_pairs_padding_via_cigarstring(self):
+ a = self.build_read()
+ a.cigarstring = "1M1P1M"
+ # The padding operation is like an insertion into the reference.
+ # See comment in test_get_aligned_pairs_padding_with_seq (below).
+ self.check_get_aligned_pairs_combos_without_MD(a,
+ [(0, 20, CMATCH), (1, None, CPAD), (2, 21, CMATCH)])
def test_get_aligned_pairs_padding_with_seq(self):
a = self.build_read()
# string: "Alternatively, to describe the same alignments, we can
# modify the reference sequence to contain pads that make room for
# sequences inserted relative to the reference."
- self.assertEqual(a.get_aligned_pairs(with_seq=True),
- [(0, 20, 'A'), (1, None, None), (2, 21, 'T')])
+ self.check_get_aligned_pairs_combos(a,
+ [(0, 20, "A", CMATCH), (1, None, None, CPAD), (2, 21, "T", CMATCH)])
def test_get_aligned_pairs(self):
a = self.build_read()
a.query_sequence = "A" * 9
a.cigarstring = "9M"
a.set_tag("MD", "9")
- self.assertEqual(
- a.get_aligned_pairs(with_seq=True),
+ self.check_get_aligned_pairs_combos(
+ a,
[
- (0, 20, "A"),
- (1, 21, "A"),
- (2, 22, "A"),
- (3, 23, "A"),
- (4, 24, "A"),
- (5, 25, "A"),
- (6, 26, "A"),
- (7, 27, "A"),
- (8, 28, "A"),
+ (0, 20, "A", CMATCH),
+ (1, 21, "A", CMATCH),
+ (2, 22, "A", CMATCH),
+ (3, 23, "A", CMATCH),
+ (4, 24, "A", CMATCH),
+ (5, 25, "A", CMATCH),
+ (6, 26, "A", CMATCH),
+ (7, 27, "A", CMATCH),
+ (8, 28, "A", CMATCH),
],
)
a.set_tag("MD", "4C4")
- self.assertEqual(
- a.get_aligned_pairs(with_seq=True),
+ self.check_get_aligned_pairs_combos(
+ a,
[
- (0, 20, "A"),
- (1, 21, "A"),
- (2, 22, "A"),
- (3, 23, "A"),
- (4, 24, "c"),
- (5, 25, "A"),
- (6, 26, "A"),
- (7, 27, "A"),
- (8, 28, "A"),
+ (0, 20, "A", CMATCH),
+ (1, 21, "A", CMATCH),
+ (2, 22, "A", CMATCH),
+ (3, 23, "A", CMATCH),
+ (4, 24, "c", CMATCH),
+ (5, 25, "A", CMATCH),
+ (6, 26, "A", CMATCH),
+ (7, 27, "A", CMATCH),
+ (8, 28, "A", CMATCH),
],
)
a.cigarstring = "5M2D4M"
a.set_tag("MD", "4C^TT4")
- self.assertEqual(
- a.get_aligned_pairs(with_seq=True),
+ self.check_get_aligned_pairs_combos(
+ a,
[
- (0, 20, "A"),
- (1, 21, "A"),
- (2, 22, "A"),
- (3, 23, "A"),
- (4, 24, "c"),
- (None, 25, "T"),
- (None, 26, "T"),
- (5, 27, "A"),
- (6, 28, "A"),
- (7, 29, "A"),
- (8, 30, "A"),
+ (0, 20, "A", CMATCH),
+ (1, 21, "A", CMATCH),
+ (2, 22, "A", CMATCH),
+ (3, 23, "A", CMATCH),
+ (4, 24, "c", CMATCH),
+ (None, 25, "T", CDEL),
+ (None, 26, "T", CDEL),
+ (5, 27, "A", CMATCH),
+ (6, 28, "A", CMATCH),
+ (7, 29, "A", CMATCH),
+ (8, 30, "A", CMATCH),
],
)
a.cigarstring = "5M2D2I2M"
a.set_tag("MD", "4C^TT2")
- self.assertEqual(
- a.get_aligned_pairs(with_seq=True),
+ self.check_get_aligned_pairs_combos(
+ a,
[
- (0, 20, "A"),
- (1, 21, "A"),
- (2, 22, "A"),
- (3, 23, "A"),
- (4, 24, "c"),
- (None, 25, "T"),
- (None, 26, "T"),
- (5, None, None),
- (6, None, None),
- (7, 27, "A"),
- (8, 28, "A"),
+ (0, 20, "A", CMATCH),
+ (1, 21, "A", CMATCH),
+ (2, 22, "A", CMATCH),
+ (3, 23, "A", CMATCH),
+ (4, 24, "c", CMATCH),
+ (None, 25, "T", CDEL),
+ (None, 26, "T", CDEL),
+ (5, None, None, CINS),
+ (6, None, None, CINS),
+ (7, 27, "A", CMATCH),
+ (8, 28, "A", CMATCH),
],
)
a.cigarstring = "5M1N5M"
a.set_tag("MD", "10")
- self.assertEqual(
- a.get_aligned_pairs(with_seq=True),
- [
- (0, 20, "A"),
- (1, 21, "A"),
- (2, 22, "A"),
- (3, 23, "A"),
- (4, 24, "A"),
- (None, 25, None),
- (5, 26, "A"),
- (6, 27, "A"),
- (7, 28, "A"),
- (8, 29, "A"),
- (9, 30, "A"),
- ],
- )
-
- self.assertEqual(
- a.get_aligned_pairs(with_seq=False),
- [
- (0, 20),
- (1, 21),
- (2, 22),
- (3, 23),
- (4, 24),
- (None, 25),
- (5, 26),
- (6, 27),
- (7, 28),
- (8, 29),
- (9, 30),
- ],
- )
-
- self.assertEqual(
- a.get_aligned_pairs(matches_only=True, with_seq=False),
+ self.check_get_aligned_pairs_combos(
+ a,
[
- (0, 20),
- (1, 21),
- (2, 22),
- (3, 23),
- (4, 24),
- (5, 26),
- (6, 27),
- (7, 28),
- (8, 29),
- (9, 30),
- ],
+ (0, 20, "A", CMATCH),
+ (1, 21, "A", CMATCH),
+ (2, 22, "A", CMATCH),
+ (3, 23, "A", CMATCH),
+ (4, 24, "A", CMATCH),
+ (None, 25, None, CREF_SKIP),
+ (5, 26, "A", CMATCH),
+ (6, 27, "A", CMATCH),
+ (7, 28, "A", CMATCH),
+ (8, 29, "A", CMATCH),
+ (9, 30, "A", CMATCH),
+ ]
)
def test_equivalence_matches_only_and_with_seq(self):
a.query_sequence = "ACGT" * 2
a.cigarstring = "4M1D4M"
a.set_tag("MD", "4^x4")
- full = (
- list(zip(range(0, 4), range(20, 24), "ACGT"))
- + [(None, 24, "x")]
- + list(zip(range(4, 8), range(25, 29), "ACGT"))
- )
- self.assertEqual(a.get_aligned_pairs(matches_only=False, with_seq=True), full)
-
- self.assertEqual(
- a.get_aligned_pairs(matches_only=True, with_seq=True),
- [x for x in full if x[0] is not None and x[1] is not None],
+ self.check_get_aligned_pairs_combos(
+ a,
+ list(zip(range(0, 4), range(20, 24), "ACGT", [CMATCH] * 4))
+ + [(None, 24, "x", CDEL)]
+ + list(zip(range(4, 8), range(25, 29), "ACGT", [CMATCH] * 4)),
)
a = self.build_read()
a.query_sequence = "ACGT" * 2
a.cigarstring = "4M1N4M"
a.set_tag("MD", "8")
- full = (
- list(zip(range(0, 4), range(20, 24), "ACGT"))
- + [(None, 24, None)]
- + list(zip(range(4, 8), range(25, 29), "ACGT"))
- )
- self.assertEqual(a.get_aligned_pairs(matches_only=False, with_seq=True), full)
-
- self.assertEqual(
- a.get_aligned_pairs(matches_only=True, with_seq=True),
- [x for x in full if x[0] is not None and x[1] is not None],
+ self.check_get_aligned_pairs_combos(
+ a,
+ list(zip(range(0, 4), range(20, 24), "ACGT", [CMATCH] * 4))
+ + [(None, 24, None, 3)]
+ + list(zip(range(4, 8), range(25, 29), "ACGT", [CMATCH] * 4)),
)
def test_get_aligned_pairs_lowercase_md(self):
expect = {
("C", 0, "m"): [(6, 102), (17, 128), (20, 153), (31, 179), (34, 204)],
("N", 0, "n"): [(15, 212)],
- ("C", 0, 76792): [(19, 161), (34, 187)],
+ ("C", 0, 76792): [(19, 161), (34, 33)],
}
with pysam.AlignmentFile(filename, check_sq=False) as inf:
r = next(iter(inf))
self.assertDictEqual(r.modified_bases, expect)
+ def testExplicit(self):
+ """reference bases should always be the same nucleotide
+ """
+ filename = os.path.join(BAM_DATADIR, "MM-explicit.bam")
+ expected_output = [
+ {("C", 0, "m"): [(9, 200), (10, 50), (14, 160)], ("C", 0, "h"): [(9, 10), (10, 170), (14, 20)]},
+ {("C", 0, "m"): [(9, 200), (10, 50), (13, 10), (14, 160), (16, 10)],
+ ("C", 0, "h"): [(9, 10), (10, 170), (13, 5), (14, 20), (16, 5)]},
+ {("C", 0, "m"): [(9, 200), (14, 160)], ("C", 0, "h"): [(9, 10), (10, 170), (13, 5), (14, 20), (16, 5)]},
+ ]
+
+ with pysam.AlignmentFile(filename, check_sq=False) as inf:
+ for r, expected in zip(inf, expected_output):
+ self.assertDictEqual(r.modified_bases, expected)
+
def testMulti(self):
"""reference bases should always be the same nucleotide
"""
alt_value_type="I",
)
+ def test_set_tag_invalid_value_type(self):
+ with self.assertRaises(ValueError):
+ self.check_tag("TT", "abc", value_type="#")
+
+ def test_set_array_tag_invalid_value_type(self):
+ with self.assertRaises(ValueError):
+ self.check_tag("TT", array.array('I', range(4)), value_type='#')
+
+ def test_set_array_tag_invalid_typecode(self):
+ with self.assertRaises(ValueError):
+ self.check_tag("TT", array.array('L', range(4)), value_type=None)
+
class TestSetTagsGetTag(TestSetTagGetTag):
def check_tag(self, tag, value, value_type, alt_value_type=None):
def check_read_write(self, flag_write, header):
fn = get_temp_filename()
- print(fn)
with pysam.AlignmentFile(
fn,
flag_write,
reference_filename=os.path.join(BAM_DATADIR, "ex1.fa")) as outf:
a = pysam.AlignedSegment()
a.query_name = "abc"
+ a.flag = pysam.FUNMAP
outf.write(a)
with pysam.AlignmentFile(fn) as inf:
read_header = inf.header
- # os.unlink(fn)
+ os.unlink(fn)
self.compare_headers(header, read_header)
expected_lengths = dict([(x["SN"], x["LN"]) for x in header["SQ"]])
self.assertEqual(expected_lengths,
import inspect
import subprocess
import tempfile
+import time
+
import pysam
WORKDIR = os.path.abspath(os.path.join(os.path.dirname(__file__),
def make_data_files(directory):
- what = None
+ if os.path.exists(os.path.join(directory, 'all.stamp')):
+ return
+
+ make = os.environ.get('MAKE', 'make')
+
+ for attempt in range(1, 6):
+ try:
+ os.mkdir(os.path.join(directory, 'all.lock'), 0o700)
+ break
+ except FileExistsError:
+ time.sleep(attempt)
+ continue
+ else:
+ raise RuntimeError(f'Directory {directory!r} already locked: try `{make} clean` there')
+
try:
- if not os.path.exists(os.path.join(directory, "all.stamp")):
- subprocess.check_output(["make", "-C", directory], stderr=subprocess.STDOUT)
+ subprocess.check_output([make, '-C', directory], stderr=subprocess.STDOUT, encoding='ascii')
except subprocess.CalledProcessError as e:
- what = "Making test data in '%s' failed:\n%s" % (directory, force_str(e.output))
-
- if what is not None:
- raise RuntimeError(what)
+ raise RuntimeError(f'Making test data in {directory!r} failed:\n{e.output}') from None
+ finally:
+ os.rmdir(os.path.join(directory, 'all.lock'))
def load_and_convert(filename, encode=True):
fn_small = "small.vcf.gz"
if not os.path.exists(fn_small):
os.system("bcftools view {} | head -n 10000 | bgzip > {}".format(fn, fn_small))
- os.system("tabix -p vcf {}".format(fn_small))
+ os.system("tabix -f -p vcf {}".format(fn_small))
return fn_small
with pytest.raises(ValueError, match='Use .allele_indices to set integer allele indices'):
record.samples['sample1'].alleles = (1, 0)
+
+
+def test_repeated_new_record(vcf_header):
+ vcf_header.formats.add('GT', 1, 'String', "Genotype")
+ vcf_header.formats.add("AA", 1, "String", "An annotation")
+ vcf_header.formats.add("BB", 1, "String", "Another annotation")
+
+ data = {'id': 'INS_1', 'contig': '1', 'start': 10, 'stop': 15, 'alleles': ['A', 'TCGA'],
+ 'samples': [{'AA': ('one'), 'GT': (0, 1), 'BB': ('two')},
+ {'GT': (1, 0), 'BB': ('three')}]}
+
+ record1 = vcf_header.new_record(**data)
+ assert '\tGT:' in str(record1) # Verify that GT is output first
+ assert record1.samples['sample1'].alleles == ('A', 'TCGA')
+ assert record1.samples['sample2'].alleles == ('TCGA', 'A')
+
+ record2 = vcf_header.new_record(**data)
+ assert '\tGT:' in str(record2) # Verify that GT is actually emitted and is output first
+ assert record2.samples['sample1'].alleles == ('A', 'TCGA')
+ assert record2.samples['sample2'].alleles == ('TCGA', 'A')
%.vcf.gz: %.vcf
bgzip < $< > $@
- tabix -p vcf $@ # create tbi index
- bcftools index $@ # create csi index
+ tabix -f -p vcf $@ # create tbi index
+ bcftools index -f $@ # create csi index
%.bcf: %.vcf.gz
bcftools view -O b $< -o $@
- bcftools index $@
+ bcftools index -f $@
example_empty.bcf: example_empty.vcf.gz
touch $@
clean:
- -rm -f all.stamp *.gz *.tbi *.csi *.bcf
+ -rm -rf all.lock all.stamp *.gz *.tbi *.csi *.bcf
@CO Separate m, h and N modifications
-* 0 * 0 0 * * 0 0 AGCTCTCCAGAGTCGNACGCCATYCGCGCGCCACCA * Mm:Z:C+m,2,2,1,4,1;C+76792,6,7;N+n,15; Ml:B:C,102,128,153,179,204,161,187,212,169
+* 0 * 0 0 * * 0 0 AGCTCTCCAGAGTCGNACGCCATYCGCGCGCCACCA * Mm:Z:C+m,2,2,1,4,1;C+76792,6,7;N+n,15; Ml:B:C,102,128,153,179,204,161,33,212
--- /dev/null
+@CO Testing explicit vs implicit base modifications.
+@CO This covers the case where a lack of a signal could be either
+@CO implicitly assumed to be no-mod (default) or assumed to be
+@CO unchecked and require an explicit statement to indicate it was
+@CO looked at and no base modification was observed.
+@CO
+@CO ATCATCATTCCTACCGCTATAGCCT r1; implicit
+@CO - - .. -. - --
+@CO Mm M
+@CO - - .. -. - --
+@CO hH h
+@CO
+@CO ATCATCATTCCTACCGCTATAGCCT r2; explicit to a small region
+@CO - - ?? ?? ? --
+@CO Mm mM m
+@CO - - ?? ?? ? --
+@CO hH hh h
+@CO
+@CO ATCATCATTCCTACCGCTATAGCCT r3; mixture
+@CO - - . -. - --
+@CO M M
+@CO - - ?? ?? ? --
+@CO hH hh h --
+@CO
+r1 0 * 0 0 * * 0 0 ATCATCATTCCTACCGCTATAGCCT * Mm:Z:C+mh,2,0,1; Ml:B:C,200,10,50,170,160,20
+r2 0 * 0 0 * * 0 0 ATCATCATTCCTACCGCTATAGCCT * Mm:Z:C+mh?,2,0,0,0,0; Ml:B:C,200,10,50,170,10,5,160,20,10,5
+r3 0 * 0 0 * * 0 0 ATCATCATTCCTACCGCTATAGCCT * Mm:Z:C+m.,2,2;C+h?,2,0,0,0,0; Ml:B:C,200,160,10,170,5,20,5
@CO r2 has them combined together, for example as produced by
@CO a joint basecaller which assigns probabilities to all
@CO trained events simultaneously.
-r1 0 * 0 0 * * 0 0 AGCTCTCCAGAGTCGNACGCCATYCGCGCGCCACCA * Mm:Z:C+m,2,2,1,4,1;C+h,6,7;N+n,15,2; Ml:B:C,128,153,179,204,230,159,6,215,240
+r1 0 * 0 0 * * 0 0 AGCTCTCCAGAGTCGNACGCCATYCGCGCGCCACCA * Mm:Z:C+m,2,2,1,4,1;C+h,6,7;N+n,15,2; Ml:B:C,128,153,179,204,230,159,6,215,240 MN:i:36
r2 0 * 0 0 * * 0 0 AGCTCTCCAGAGTCGNACGCCATYCGCGCGCCACCA * Mm:Z:C+mh,2,2,0,0,4,1;N+n,15; Ml:B:C,77,159,103,133,128,108,154,82,179,57,204,31,240
cp ex1.cram $@
clean:
- rm -fr [a-z]*.bam *.bai *.csi *.fai *.gzi *.pileup* [a-z]*.cram *.crai \
- all.stamp *~ calDepth *.dSYM pysam_*.sam \
+ rm -fr [A-Za-z]*.bam *.bai *.csi *.fai *.gzi *.pileup* [A-Za-z]*.cram *.crai \
+ all.lock all.stamp *~ calDepth *.dSYM pysam_*.sam \
ex2.sam ex2.sam.gz ex1.sam ex1.fa.gz \
with_md.sam.gz \
*.fq.gz
"index ex1.bam %(out)s_ex1.bam.fai",
"index -@2 ex1.bam %(out)s_ex1.bam.fai",
"idxstats ex1.bam > %(out)s_ex1.idxstats",
- "fixmate ex1.bam %(out)s_ex1.fixmate.bam",
+ # TODO: fixmate behaviour changed in 1.21
+ #"fixmate ex1.bam %(out)s_ex1.fixmate.bam",
"flagstat ex1.bam > %(out)s_ex1.flagstat",
"calmd ex1.bam ex1.fa > %(out)s_ex1.calmd.bam",
# use -s option, otherwise the following error in samtools 1.2:
command = self.get_command(statement)
- # self.assertTrue(command in pysam.SAMTOOLS_DISPATCH)
-
targets = [x for x in parts if "%(out)s" in x]
samtools_targets = [x % r_samtools for x in targets]
pysam_targets = [x % r_pysam for x in targets]
bgzip -c $< > $@
%.gff2.gz.tbi: %.gff2.gz
- tabix -p gff $<
+ tabix -f -p gff $<
%.gff3.gz.tbi: %.gff3.gz
- tabix -p gff $<
+ tabix -f -p gff $<
%.gtf.gz.tbi: %.gtf.gz
- tabix -p gff $<
+ tabix -f -p gff $<
%.gz.tbi: %.gz
- tabix -p $(subst .,,$(suffix $*)) $<
+ tabix -f -p $(subst .,,$(suffix $*)) $<
clean:
- -rm -f all.stamp $(DERIVED_FILES)
+ -rm -rf all.lock all.stamp $(DERIVED_FILES)
--- /dev/null
+import inspect
+import os
+import pytest
+import re
+from typing import TYPE_CHECKING
+
+import pysam
+import pysam.samtools
+import pysam.bcftools
+
+from TestUtils import BAM_DATADIR, make_data_files
+
+try:
+ import mypy.api
+except ImportError:
+ pytest.skip('mypy API not available', allow_module_level=True)
+
+PREAMBLE = """
+import re
+from typing import TYPE_CHECKING
+
+import pysam
+import pysam.samtools
+import pysam.bcftools
+
+def typecheck(check_locals = True): pass
+"""
+
+MYPY_OPTIONS = ['--no-incremental', '--no-error-summary', '--follow-imports', 'silent']
+
+
+def typecheck(check_locals: bool = True):
+ myframe = inspect.currentframe()
+ if not myframe: pytest.skip('current stack frame not available')
+ caller = myframe.f_back
+ if not caller: pytest.skip('caller stack frame not available')
+
+ code = inspect.getsource(caller)
+ stdout, stderr, status = mypy.api.run(MYPY_OPTIONS + ['--command', PREAMBLE + code])
+ assert status == 0, f'mypy failed:\n{stdout}{stderr}'
+
+ types = {}
+ for line in stdout.splitlines():
+ m = re.search(r'note: *(\w+): ([\w.]*)', line)
+ if m: types[m.group(1)] = m.group(2)
+
+ def _plain(s):
+ s = re.sub(r"<class '([^']*)'>", r'\1', s)
+ s = re.sub(r'builtins\.', '', s)
+ return s
+
+ if check_locals:
+ for var, vartype in types.items():
+ assert _plain(vartype) == _plain(repr(type(caller.f_locals[var]))), f'Incorrect type for {var!r}'
+
+ return types
+
+
+def setUpModule():
+ make_data_files(BAM_DATADIR)
+
+
+@pytest.fixture
+def aln():
+ header = pysam.AlignmentHeader.from_references(['chr1', 'chr2'], [50000, 20000])
+ a = pysam.AlignedSegment(header)
+ a.query_name = 'read_one'
+ a.flag = pysam.FPAIRED | pysam.FREAD1
+ a.reference_id = 0
+ a.reference_start = 1000
+ a.mapping_quality = 20
+ a.next_reference_id = 1
+ a.next_reference_start = 5000
+ a.template_length = 0
+ a.cigartuples = ((pysam.CMATCH, 6), (pysam.CINS, 4), (pysam.CMATCH, 2))
+ a.query_sequence = 'ATGCATGCATGC'
+ a.query_qualities = pysam.qualitystring_to_array('abcdefghijkl')
+ return a
+
+
+@pytest.fixture
+def sam_fname():
+ return os.path.join(BAM_DATADIR, 'ex3.sam')
+
+
+@pytest.fixture
+def bam_fname():
+ return os.path.join(BAM_DATADIR, 'ex1.bam')
+
+
+def test_AlignedSegment_is_mapped(aln: pysam.AlignedSegment) -> None:
+ rmap = aln.is_mapped
+ mmap = aln.mate_is_mapped
+ rfwd = aln.is_forward
+ mfwd = aln.mate_is_forward
+
+ if TYPE_CHECKING: reveal_locals()
+ types = typecheck()
+
+
+def test_AlignmentHeader_get_reference_length(sam_fname: str) -> None:
+ inf = pysam.AlignmentFile(sam_fname)
+ n1 = inf.get_reference_length('chr1')
+ hdr = inf.header
+ n2 = hdr.get_reference_length('chr1')
+
+ if TYPE_CHECKING: reveal_locals()
+ types = typecheck()
+
+
+def test_pileup_iterator_column(bam_fname: str) -> None:
+ inf = pysam.AlignmentFile(bam_fname)
+ pu = inf.pileup('chr1')
+ for p in pu:
+ pid = p.reference_id
+ ppos = p.reference_pos
+
+ if TYPE_CHECKING: reveal_locals()
+ types = typecheck(check_locals=False)
+ assert re.search(r'\.IteratorColumn(|All|AllRefs|Region)$', types['pu'])
+ assert types['p'].endswith('PileupColumn')
+ assert types['pid'] == 'builtins.int'
+ assert types['ppos'] == 'builtins.int'
+
+
+def test_samtools_subcommands() -> None:
+ p1_samtools_faidx = pysam.samtools.faidx
+ p2_faidx = pysam.faidx
+ p3_view = pysam.view
+ p4_bcftools_view = pysam.bcftools.view
+
+ if TYPE_CHECKING: reveal_locals()
+ types = typecheck()
+ for var, vartype in types.items():
+ assert vartype.endswith('PysamDispatcher'), f'{var!r} is not a dispatcher'
projects / python-pysam.git / commitdiff