--- /dev/null
--- /dev/null
++# senencepiece for Debian
++
++The upstream of sentencepiece 0.1.97 was initially released around June 6, 2022,
++but it was withdrawed and re-released as same version at Aug 7, 2022 again.
++
++Thus, some commits were not included into 0.1.97-1.
++
++To fix up this issue, commits since 5e5adf2f851a1514ccc435aae11ee830c438321b
++were applied as the following patch files.
++
++NOTE: Drop these patch files when newer version was released.
++
++0001-update-python-wrapper.patch
++0002-remove-debug-symbols-from-wheel-package.patch
++0003-allow-tab-character-to-be-used-in-user_defined_symbo.patch
++0004-add-test-to-use-tab-as-user-defined-symbols.patch
++0005-Uses-C-17-by-default.patch
++0006-Uses-std-atomic-to-define-global-variable.patch
++0007-Fix-a-typo.patch
++0008-Uses-absl-string_view-as-much-as-possible.patch
++0009-Fixed-build-break.patch
++0010-Added-ImmutableSentencePiece-class.patch
++0011-add-verbose-option.patch
++0012-Supports-ImmutableSentencePieceText-from-python-modu.patch
++0013-Adds-more-unittests.patch
++0014-Adds-SWIGPYTHON-flag.patch
++0015-remove-unused-ifdef-SWIG-macro.patch
++0016-Fixed-test-failure.patch
++0017-Uses-property-in-immutable-proto.patch
++0018-automatically-detect-the-number-of-CPUs-in-batch-pro.patch
++0019-support-slice-in-pieces-nbests-objects.patch
++0020-Updated-the-document.patch
++0021-Fixed-errors-in-example-notebook.patch
++0022-Fix-dead-links.patch
++0023-added-ShutdownLibrary-function-to-uninitialize-globa.patch
++0024-Fixed-the-issue-of-concatinating-paths-for-pkg-confi.patch
++
++
--- /dev/null
--- /dev/null
++sentencepiece (0.1.97-3) unstable; urgency=medium
++
++ * debian/patches/0001-update-python-wrapper.patch
++ debian/patches/0002-remove-debug-symbols-from-wheel-package.patch
++ debian/patches/0003-allow-tab-character-to-be-used-in-user_defined_symbo.patch
++ debian/patches/0004-add-test-to-use-tab-as-user-defined-symbols.patch
++ debian/patches/0005-Uses-C-17-by-default.patch
++ debian/patches/0006-Uses-std-atomic-to-define-global-variable.patch
++ debian/patches/0007-Fix-a-typo.patch
++ debian/patches/0008-Uses-absl-string_view-as-much-as-possible.patch
++ debian/patches/0009-Fixed-build-break.patch
++ debian/patches/0010-Added-ImmutableSentencePiece-class.patch
++ debian/patches/0011-add-verbose-option.patch
++ debian/patches/0012-Supports-ImmutableSentencePieceText-from-python-modu.patch
++ debian/patches/0013-Adds-more-unittests.patch
++ debian/patches/0014-Adds-SWIGPYTHON-flag.patch
++ debian/patches/0015-remove-unused-ifdef-SWIG-macro.patch
++ debian/patches/0016-Fixed-test-failure.patch
++ debian/patches/0017-Uses-property-in-immutable-proto.patch
++ debian/patches/0018-automatically-detect-the-number-of-CPUs-in-batch-pro.patch
++ debian/patches/0019-support-slice-in-pieces-nbests-objects.patch
++ debian/patches/0020-Updated-the-document.patch
++ debian/patches/0021-Fixed-errors-in-example-notebook.patch
++ debian/patches/0022-Fix-dead-links.patch
++ debian/patches/0023-added-ShutdownLibrary-function-to-uninitialize-globa.patch
++ debian/patches/0024-Fixed-the-issue-of-concatinating-paths-for-pkg-confi.patch
++ - Add missing patch files for 0.1.97.
++ * debian/README.Debian
++ - Add explanation of debian/patches.
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Mon, 21 Nov 2022 22:43:46 +0900
++
++sentencepiece (0.1.97-2) unstable; urgency=medium
++
++ * Team upload
++
++ [ Steve Langasek ]
++ * debian/patches/header-dependencies.patch: include necessary headers
++ to ensure IS_BIG_ENDIAN is defined, see #1017360.
++
++ -- Graham Inggs <ginggs@debian.org> Sun, 18 Sep 2022 05:30:57 +0000
++
++sentencepiece (0.1.97-1) unstable; urgency=medium
++
++ * New upstream version 0.1.97
++ * debian/copyright
++ - Update maintainer E-mail address
++ * debian/control
++ - Bump Standards-Version to 4.6.1. No other changes are required.
++ * debian/patches/support-python-module-in-place.patch
++ - Refresh path to build python module.
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Tue, 14 Jun 2022 20:19:58 +0900
++
++sentencepiece (0.1.96-1) unstable; urgency=medium
++
++ * New upstream version 0.1.96
++ * debian/control
++ - Bump standard-version to 4.5.1. No changes are required.
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Wed, 18 Aug 2021 20:52:46 +0900
++
++sentencepiece (0.1.95-1) unstable; urgency=medium
++
++ * New upstream version 0.1.95
++ * debian/patches/support-python-module-in-place.patch
++ - Fix undefined symbol when importing python module (Closes: #979040)
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Thu, 11 Feb 2021 17:36:23 +0900
++
++sentencepiece (0.1.94-2) unstable; urgency=medium
++
++ * Fix FTBFS on armel/mipsel (Closes: #977235)
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Wed, 16 Dec 2020 21:18:15 +0900
++
++sentencepiece (0.1.94-1) unstable; urgency=medium
++
++ * New upstream version 0.1.94
++ * debian/patches/support-python-module-in-place.patch
++ - Refresh path to build python module.
++ * debian/patches/fix-ftbfs-ports.patch
++ debian/patches/mutiarch-support.patch
++ - Remove needless patch because these patch was merged
++ to google/sentencepiece.
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Wed, 28 Oct 2020 21:02:07 +0900
++
++sentencepiece (0.1.93-1) unstable; urgency=medium
++
++ * New upstream version 0.1.93
++ * debian/source/lintian-overrides
++ - Remove needless override.
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Thu, 15 Oct 2020 21:32:05 +0900
++
++sentencepiece (0.1.92-3) unstable; urgency=medium
++
++ * debian/patches/fix-ftbfs-ports.patch
++ - Fix FTBFS on powerpc
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Sat, 03 Oct 2020 20:48:27 +0900
++
++sentencepiece (0.1.92-2) unstable; urgency=medium
++
++ * debian/patches/0002-Change-in-order-to-build-Python-modules-in-place.patch
++ - Fix FTBFS on hurd-i386
++ * debian/patches/0004-Fix-FTBFS-on-armel-and-mipsel.patch
++ - Fix missing dependency to atomic library (powerpc,m68k,sh4)
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Sat, 26 Sep 2020 20:27:17 +0900
++
++sentencepiece (0.1.92-1) unstable; urgency=medium
++
++ * New upstream version 0.1.92
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Fri, 19 Jun 2020 19:38:49 +0900
++
++sentencepiece (0.1.91-1) unstable; urgency=medium
++
++ * New upstream version 0.1.91
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Fri, 22 May 2020 15:17:42 +0900
++
++sentencepiece (0.1.90-3) unstable; urgency=medium
++
++ * debian/patches/0004-Fix-FTBFS-on-armel-and-mipsel.patch
++ - Refresh patch to fix FTBFS.
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Sun, 17 May 2020 09:02:23 +0900
++
++sentencepiece (0.1.90-2) unstable; urgency=medium
++
++ * debian/patches/0004-Fix-FTBFS-on-armel-and-mipsel.patch
++ - Add patch to fix FTBFS on mipsel and armel
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Sat, 16 May 2020 16:16:45 +0900
++
++sentencepiece (0.1.90-1) unstable; urgency=medium
++
++ * New upstream version 0.1.90
++ * debian/control
++ - Update Uploaders:
++ - Bump standard-version to 4.5.0
++ - Bump compat version to 13.
++ * debian/source/lintian-overrides
++ - Fix false positive source-is-missing
++ * debian/patches/0003-Disable-static-library-explicitly.patch
++ - Disable to build static library
++
++ -- Kentaro Hayashi <kenhys@xdump.org> Wed, 13 May 2020 19:09:34 +0900
++
++sentencepiece (0.1.84-1) unstable; urgency=medium
++
++ * New upstream version 0.1.84 (Closes: #939860)
++
++ [ TSUCHIYA Masatoshi ]
++ * Initial packaging tasks.
++ * Remove pipeline configurations for BitBucket.
++
++ [ Kentaro Hayashi ]
++ * debian/gbp.conf
++ - Add basic configuration about debian-branch
++ * debian/watch
++ - Add missing watch file to detect a new release
++ * debian/control
++ - Update deprecated Priority: to optional
++ - Add Vcs-* fields
++ - Fix W: sentencepiece: description-synopsis-starts-with-article
++ - Bump standard version to 4.4.1
++ - Update Vcs-* under science-team
++ - Bump up compatibility level
++ - Drop python2 support
++ * debian/copyright
++ - Use https://
++ - Update copyright about third party modules
++ * debian/rules
++ - Enable hardening
++ * debian/salsa-ci.yml
++ - Add Salsa CI configuration
++
++ -- Kentaro Hayashi <hayashi@clear-code.com> Thu, 17 Oct 2019 13:33:34 +0900
--- /dev/null
--- /dev/null
++Source: sentencepiece
++Section: science
++Priority: optional
++Maintainer: Debian Science Maintainers <debian-science-maintainers@lists.alioth.debian.org>
++Uploaders:
++ TSUCHIYA Masatoshi <tsuchiya@namazu.org>,
++ Kentaro Hayashi <kenhys@xdump.org>
++Build-Depends:
++ debhelper-compat (= 13),
++ protobuf-compiler,
++ libprotobuf-dev,
++ dh-python,
++ python3-all-dev,
++ quilt,
++ cmake,
++ python3-setuptools
++Standards-Version: 4.6.1
++Homepage: https://github.com/google/sentencepiece
++Vcs-Browser: https://salsa.debian.org/science-team/sentencepiece
++Vcs-Git: https://salsa.debian.org/science-team/sentencepiece.git
++Rules-Requires-Root: no
++
++Package: sentencepiece
++Architecture: any
++Depends: ${shlibs:Depends}, ${misc:Depends}
++Description: Unsupervised text tokenizer and detokenizer
++ SentencePiece is an unsupervised text tokenizer/detokenizer mainly
++ designed for Neural Network-based text generation systems where the
++ vocabulary size is predetermined prior to the neural model training.
++
++Package: libsentencepiece0
++Section: libs
++Architecture: any
++Depends: ${shlibs:Depends}, ${misc:Depends}
++Description: Library files of SentencePiece
++ SentencePiece is an unsupervised text tokenizer/detokenizer mainly
++ designed for Neural Network-based text generation systems where the
++ vocabulary size is predetermined prior to the neural model training.
++
++Package: libsentencepiece-dev
++Section: libdevel
++Architecture: any
++Depends: libsentencepiece0 (= ${binary:Version}), ${misc:Depends}
++Description: Header files of SentencePiece
++ SentencePiece is an unsupervised text tokenizer/detokenizer mainly
++ designed for Neural Network-based text generation systems where the
++ vocabulary size is predetermined prior to the neural model training.
++
++Package: python3-sentencepiece
++Section: python
++Architecture: any
++Depends:
++ ${shlibs:Depends},
++ ${misc:Depends},
++ ${python3:Depends}
++Description: SentencePiece binding for Python3
++ SentencePiece is an unsupervised text tokenizer/detokenizer mainly
++ designed for Neural Network-based text generation systems where the
++ vocabulary size is predetermined prior to the neural model training.
++ .
++ python3-sentencepiece is its binding for Python3.
--- /dev/null
--- /dev/null
++Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
++Upstream-Name: sentencepiece
++Source: https://github.com/google/sentencepiece
++
++Files: *
++Copyright: 2017 Taku Kudo <taku@chasen.org>
++License: Apache-2.0
++ Licensed under the Apache License, Version 2.0 (the "License");
++ you may not use this file except in compliance with the License.
++ You may obtain a copy of the License at
++ .
++ http://www.apache.org/licenses/LICENSE-2.0
++ .
++ Unless required by applicable law or agreed to in writing, software
++ distributed under the License is distributed on an "AS IS" BASIS,
++ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
++ implied. See the License for the specific language governing
++ permissions and limitations under the License.
++
++Files: debian/*
++Copyright:
++ 2016 TSUCHIYA Masatoshi <tsuchiya@namazu.org>
++ 2019-2022 Kentaro Hayashi <kenhys@xdump.org>
++License: GPL-2+
++ This package is free software; you can redistribute it and/or modify
++ it under the terms of the GNU General Public License as published by
++ the Free Software Foundation; either version 2 of the License, or
++ (at your option) any later version.
++ .
++ This package is distributed in the hope that it will be useful,
++ but WITHOUT ANY WARRANTY; without even the implied warranty of
++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
++ GNU General Public License for more details.
++ .
++ You should have received a copy of the GNU General Public License
++ along with this program. If not, see <http://www.gnu.org/licenses/>
++ .
++ On Debian systems, the complete text of the GNU General
++ Public License version 2 can be found in "/usr/share/common-licenses/GPL-2".
++
++Files: third_party/esaxx/*
++Copyright: 2010 Daisuke Okanohara
++License: MIT
++
++Files: third_party/darts_clone/*
++Copyright: 2008-2011, Susumu Yata
++License: BSD-3-clause
++
++Files: third_party/protobuf-lite/*
++Copyright: 2008 Google Inc.
++License: BSD-3-clause
++
++Files: data/Scripts.txt
++Copyright: 1991-2016 Unicode, Inc.
++License: Unicode
++ COPYRIGHT AND PERMISSION NOTICE
++ .
++ Copyright © 1991-2016 Unicode, Inc. All rights reserved.
++ Distributed under the Terms of Use in https://www.unicode.org/copyright.html.
++ .
++ Permission is hereby granted, free of charge, to any person obtaining
++ a copy of the Unicode data files and any associated documentation
++ (the "Data Files") or Unicode software and any associated documentation
++ (the "Software") to deal in the Data Files or Software
++ without restriction, including without limitation the rights to use,
++ copy, modify, merge, publish, distribute, and/or sell copies of
++ the Data Files or Software, and to permit persons to whom the Data Files
++ or Software are furnished to do so, provided that either
++ (a) this copyright and permission notice appear with all copies
++ of the Data Files or Software, or
++ (b) this copyright and permission notice appear in associated
++ Documentation.
++ .
++ THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF
++ ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
++ WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
++ NONINFRINGEMENT OF THIRD PARTY RIGHTS.
++ IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS
++ NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL
++ DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
++ DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
++ TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
++ PERFORMANCE OF THE DATA FILES OR SOFTWARE.
++ .
++ Except as contained in this notice, the name of a copyright holder
++ shall not be used in advertising or otherwise to promote the sale,
++ use or other dealings in these Data Files or Software without prior
++ written authorization of the copyright holder.
++
++Files: data/botchan.txt
++Copyright: Kin-nosuke Natsume
++License: public-domain
++ Written by Kin-nosuke Natume and put into the public domain.
++ It's transalted by Yasotaro Morri and published by Project Gutenberg.
++
++Files: data/wagahaiwa_nekodearu.txt
++Copyright: Kin-nosuke Natsume
++License: public-domain
++ Written by Kin-nosuke Natume and put into the public domain.
++ It's digitized by Aozora Bunko collabolator and published by Aozora Bunko.
++
++License: MIT
++ Permission is hereby granted, free of charge, to any person
++ obtaining a copy of this software and associated documentation
++ files (the "Software"), to deal in the Software without
++ restriction, including without limitation the rights to use,
++ copy, modify, merge, publish, distribute, sublicense, and/or sell
++ copies of the Software, and to permit persons to whom the
++ Software is furnished to do so, subject to the following
++ conditions:
++ .
++ The above copyright notice and this permission notice shall be
++ included in all copies or substantial portions of the Software.
++ .
++ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
++ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
++ OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
++ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
++ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
++ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
++ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
++ OTHER DEALINGS IN THE SOFTWARE.
++
++License: BSD-3-clause
++ Redistribution and use in source and binary forms, with or without
++ modificatio n, are permitted provided that the following conditions
++ are met:
++ .
++ - Redistributions of source code must retain the above copyright
++ notice, this list of conditions and the following disclaimer.
++ - Redistributions in binary form must reproduce the above copyright
++ notice, this list of conditions and the following disclaimer in the
++ documentation and/or other materials provided with the
++ distribution.
++ - Neither the name of the <ORGANIZATION> nor the names of its
++ contributors may be used to endorse or promote products derived
++ from this software without specific prior written permission.
++ .
++ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
++ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
++ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
++ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
++ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
++ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
++ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
++ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
++ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
++ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
++ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
++
--- /dev/null
--- /dev/null
++[DEFAULT]
++debian-branch = master
++
--- /dev/null
--- /dev/null
++usr/lib/*/lib*.so
++usr/lib/*/pkgconfig/*
++usr/include/*
--- /dev/null
--- /dev/null
++usr/lib/*/lib*.so.*
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 8 Jun 2022 02:22:21 +0900
++Subject: update python wrapper.
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/make_py_wheel.sh | 73 -
++ python/make_py_wheel_mac.sh | 89 -
++ python/once.h | 157 --
++ python/src/sentencepiece/__init__.py | 293 ++-
++ python/src/sentencepiece/sentencepiece.i | 648 +++++-
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 2383 +++++++++++++++--------
++ python/test/sentencepiece_test.py | 424 ++--
++ 7 files changed, 2575 insertions(+), 1492 deletions(-)
++ delete mode 100755 python/make_py_wheel.sh
++ delete mode 100755 python/make_py_wheel_mac.sh
++ delete mode 100644 python/once.h
++
++diff --git a/python/make_py_wheel.sh b/python/make_py_wheel.sh
++deleted file mode 100755
++index 2e123ce..0000000
++--- a/python/make_py_wheel.sh
+++++ /dev/null
++@@ -1,73 +0,0 @@
++-#!/bin/bash
++-# Copyright 2018 Google Inc.
++-#
++-# Licensed under the Apache License, Version 2.0 (the "License");
++-# you may not use this file except in compliance with the License.
++-# You may obtain a copy of the License at
++-#
++-# http://www.apache.org/licenses/LICENSE-2.0
++-#
++-# Unless required by applicable law or agreed to in writing, software
++-# distributed under the License is distributed on an "AS IS" BASIS,
++-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++-# See the License for the specific language governing permissions and
++-# limitations under the License.!
++-set -e # exit immediately on error
++-set -x # display all commands
++-
++-CMAKE_VERSION=3.12.0
++-
++-run_docker() {
++- cd `dirname $0`
++- docker pull $1
++- docker run --rm -ti --name py_sentencepiece \
++- -v `pwd`/../:/sentencepiece -w /sentencepiece/python \
++- -td $1 /bin/bash
++- docker exec py_sentencepiece bash -c "./make_py_wheel.sh native $2"
++- docker stop py_sentencepiece
++-}
++-
++-build() {
++- TRG=$1
++- rm -fr build
++- mkdir -p build
++- cd build
++-
++- # Install sentencepiece
++- cmake ../.. -DSPM_ENABLE_SHARED=OFF
++- make -j4
++- make install
++- cd ..
++-
++- for i in /opt/python/*
++- do
++- export LD_LIBRARY_PATH=/usr/local/lib:/usr/lib
++- $i/bin/python setup.py clean
++- $i/bin/python setup.py bdist
++- strip build/*/*/*.so
++- $i/bin/python setup.py bdist_wheel
++- $i/bin/python setup.py test
++- rm -fr build
++- rm -fr *.so
++- done
++-
++- cd dist
++- for i in *${TRG}.whl
++- do
++- auditwheel repair $i
++- done
++-
++- mv -f wheelhouse/*${TRG}.whl .
++-
++- cd ..
++- rm -fr build
++-}
++-
++-if [ "$1" = "native" ]; then
++- build $2
++-elif [ "$#" -eq 1 ]; then
++- run_docker quay.io/pypa/manylinux2014_${1} ${1}
++-else
++- run_docker quay.io/pypa/manylinux2014_i686 i686
++- run_docker quay.io/pypa/manylinux2014_x86_64 x86_64
++-fi
++diff --git a/python/make_py_wheel_mac.sh b/python/make_py_wheel_mac.sh
++deleted file mode 100755
++index bed7366..0000000
++--- a/python/make_py_wheel_mac.sh
+++++ /dev/null
++@@ -1,89 +0,0 @@
++-#!/bin/bash
++-# Copyright 2018 Google Inc.
++-#
++-# Licensed under the Apache License, Version 2.0 (the "License");
++-# you may not use this file except in compliance with the License.
++-# You may obtain a copy of the License at
++-#
++-# http://www.apache.org/licenses/LICENSE-2.0
++-#
++-# Unless required by applicable law or agreed to in writing, software
++-# distributed under the License is distributed on an "AS IS" BASIS,
++-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++-# See the License for the specific language governing permissions and
++-# limitations under the License.!
++-
++-set -e # exit immediately on error
++-set -x # display all commands
++-
++-build_python() {
++- VERSION=$1
++- URL=$2
++- INSTALL_PATH="/Library/Frameworks/Python.framework/Versions/${VERSION}/bin"
++- CURRENT_PATH=${PATH}
++-
++- curl -L -o python.pkg ${URL}
++- sudo installer -pkg python.pkg -target /
++-
++- if [ -f "${INSTALL_PATH}/python3" ]; then
++- ln -s ${INSTALL_PATH}/python3 ${INSTALL_PATH}/python
++- ln -s ${INSTALL_PATH}/python3-config ${INSTALL_PATH}/python-config
++- ln -s ${INSTALL_PATH}/pip3 ${INSTALL_PATH}/pip
++- fi
++-
++- export PATH="${INSTALL_PATH}:${CURRENT_PATH}"
++- ls -l ${INSTALL_PATH}
++- which python
++- which pip
++- python --version
++- curl -L -o get-pip.py https://bootstrap.pypa.io/pip/3.6/get-pip.py
++- sudo python ./get-pip.py --no-setuptools --no-wheel --ignore-installed
++- pip install --upgrade setuptools
++- pip install wheel
++- pip install delocate
++- python setup.py clean
++- python setup.py bdist_wheel --plat-name=macosx_10_6_x86_64
++- python setup.py test
++- delocate-listdeps dist/*.whl
++- delocate-wheel -w dist/delocated_wheel dist/*.whl
++- export PATH="${CURRENT_PATH}"
++-
++- ls -l dist/delocated_wheel
++- rm -fr build
++- rm -fr *.so
++- rm -fr dist/*.whl
++- rm -fr python.pkg
++-}
++-
++-build() {
++- cd python
++- rm -fr build
++- mkdir -p build
++- cd build
++-
++- # Install sentencepiece
++- cmake ../.. -DSPM_ENABLE_SHARED=OFF -DSPM_NO_THREADLOCAL=ON
++- make -j4 VERBOSE=1
++- make install
++- cd ..
++-
++- mkdir -p dist/delocated_wheel
++-
++-# build_python 2.7 https://www.python.org/ftp/python/2.7.15/python-2.7.15-macosx10.6.pkg
++-# latest pip doesn't support Py3.4
++- # build_python 3.4 https://www.python.org/ftp/python/3.4.4/python-3.4.4-macosx10.6.pkg
++- curl -L -O https://bootstrap.pypa.io/pip/3.5/get-pip.py
++- build_python 3.5 https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg
++-
++- curl -L -O https://bootstrap.pypa.io/get-pip.py
++- build_python 3.6 https://www.python.org/ftp/python/3.6.6/python-3.6.6-macosx10.6.pkg
++- build_python 3.7 https://www.python.org/ftp/python/3.7.9/python-3.7.9-macosx10.9.pkg
++- build_python 3.8 https://www.python.org/ftp/python/3.8.6/python-3.8.6-macosx10.9.pkg
++- build_python 3.9 https://www.python.org/ftp/python/3.9.0/python-3.9.0-macosx10.9.pkg
++-
++- cd ..
++-
++- rm -fr build
++-}
++-
++-build
++diff --git a/python/once.h b/python/once.h
++deleted file mode 100644
++index fc7553a..0000000
++--- a/python/once.h
+++++ /dev/null
++@@ -1,157 +0,0 @@
++-// Protocol Buffers - Google's data interchange format
++-// Copyright 2008 Google Inc. All rights reserved.
++-// https://developers.google.com/protocol-buffers/
++-//
++-// Redistribution and use in source and binary forms, with or without
++-// modification, are permitted provided that the following conditions are
++-// met:
++-//
++-// * Redistributions of source code must retain the above copyright
++-// notice, this list of conditions and the following disclaimer.
++-// * Redistributions in binary form must reproduce the above
++-// copyright notice, this list of conditions and the following disclaimer
++-// in the documentation and/or other materials provided with the
++-// distribution.
++-// * Neither the name of Google Inc. nor the names of its
++-// contributors may be used to endorse or promote products derived from
++-// this software without specific prior written permission.
++-//
++-// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
++-// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
++-// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
++-// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
++-// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
++-// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
++-// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
++-// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
++-// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
++-// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
++-// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
++-
++-// Author: kenton@google.com (Kenton Varda)
++-//
++-// emulates google3/base/once.h
++-//
++-// This header is intended to be included only by internal .cc files and
++-// generated .pb.cc files. Users should not use this directly.
++-//
++-// This is basically a portable version of pthread_once().
++-//
++-// This header declares:
++-// * A type called ProtobufOnceType.
++-// * A macro GOOGLE_PROTOBUF_DECLARE_ONCE() which declares a variable of type
++-// ProtobufOnceType. This is the only legal way to declare such a variable.
++-// The macro may only be used at the global scope (you cannot create local or
++-// class member variables of this type).
++-// * A function GoogleOnceInit(ProtobufOnceType* once, void (*init_func)()).
++-// This function, when invoked multiple times given the same ProtobufOnceType
++-// object, will invoke init_func on the first call only, and will make sure
++-// none of the calls return before that first call to init_func has finished.
++-// * The user can provide a parameter which GoogleOnceInit() forwards to the
++-// user-provided function when it is called. Usage example:
++-// int a = 10;
++-// GoogleOnceInit(&my_once, &MyFunctionExpectingIntArgument, &a);
++-// * This implementation guarantees that ProtobufOnceType is a POD (i.e. no
++-// static initializer generated).
++-//
++-// This implements a way to perform lazy initialization. It's more efficient
++-// than using mutexes as no lock is needed if initialization has already
++-// happened.
++-//
++-// Example usage:
++-// void Init();
++-// GOOGLE_PROTOBUF_DECLARE_ONCE(once_init);
++-//
++-// // Calls Init() exactly once.
++-// void InitOnce() {
++-// GoogleOnceInit(&once_init, &Init);
++-// }
++-//
++-// Note that if GoogleOnceInit() is called before main() has begun, it must
++-// only be called by the thread that will eventually call main() -- that is,
++-// the thread that performs dynamic initialization. In general this is a safe
++-// assumption since people don't usually construct threads before main() starts,
++-// but it is technically not guaranteed. Unfortunately, Win32 provides no way
++-// whatsoever to statically-initialize its synchronization primitives, so our
++-// only choice is to assume that dynamic initialization is single-threaded.
++-
++-#ifndef GOOGLE_PROTOBUF_STUBS_ONCE_H__
++-#define GOOGLE_PROTOBUF_STUBS_ONCE_H__
++-
++-#include <sched.h>
++-#include <atomic>
++-#include <mutex>
++-#include <utility>
++-
++-namespace google {
++-namespace protobuf {
++-namespace internal {
++-
++-using once_flag = std::atomic<int>;
++-
++-template <typename Callable, typename... Args>
++-void my_call_once(once_flag& once, Callable&& fn, Args&&... args) {
++- enum CallOnceState {
++- ONCE_INIT = 0,
++- ONCE_RUNNING = 1,
++- ONCE_DONE = 2,
++- };
++-
++- int expected_state = ONCE_INIT;
++- if (once.compare_exchange_strong(expected_state, ONCE_RUNNING)) {
++- fn(std::forward<Args>(args)...);
++- once.store(ONCE_DONE);
++- return;
++- }
++-
++- if (expected_state == ONCE_DONE) {
++- return;
++- }
++-
++- while (once.load() == ONCE_RUNNING) {
++- sched_yield();
++- }
++-}
++-
++-template <typename... Args>
++-void call_once(Args&&... args) {
++- my_call_once(std::forward<Args>(args)...);
++-}
++-} // namespace internal
++-
++-// TODO(gerbens) remove this once third_party is fully extracted
++-using ProtobufOnceType = internal::once_flag;
++-
++-inline void GoogleOnceInit(ProtobufOnceType* once, void (*init_func)()) {
++- internal::my_call_once(*once, init_func);
++-}
++-
++-template <typename Arg>
++-inline void GoogleOnceInitArg(ProtobufOnceType* once, void (*init_func)(Arg*),
++- Arg* arg) {
++- internal::my_call_once(*once, init_func, arg);
++-}
++-
++-class GoogleOnceDynamic {
++- public:
++- // If this->Init() has not been called before by any thread,
++- // execute (*func_with_arg)(arg) then return.
++- // Otherwise, wait until that prior invocation has finished
++- // executing its function, then return.
++- template <typename T>
++- void Init(void (*func_with_arg)(T*), T* arg) {
++- GoogleOnceInitArg<T>(&this->state_, func_with_arg, arg);
++- }
++-
++- private:
++- ProtobufOnceType state_;
++-};
++-
++-#define GOOGLE_PROTOBUF_ONCE_TYPE ::google::protobuf::ProtobufOnceType
++-#define GOOGLE_PROTOBUF_DECLARE_ONCE(NAME) \
++- ::google::protobuf::ProtobufOnceType NAME
++-
++-} // namespace protobuf
++-} // namespace google
++-
++-#endif // GOOGLE_PROTOBUF_STUBS_ONCE_H__
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index fdb5976..cba3b70 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -87,48 +87,15 @@ class SentencePieceProcessor(object):
++ def LoadVocabulary(self, filename, threshold):
++ return _sentencepiece.SentencePieceProcessor_LoadVocabulary(self, filename, threshold)
++
++- def EncodeAsPieces(self, input):
++- return _sentencepiece.SentencePieceProcessor_EncodeAsPieces(self, input)
++-
++- def EncodeAsIds(self, input):
++- return _sentencepiece.SentencePieceProcessor_EncodeAsIds(self, input)
++-
++- def NBestEncodeAsPieces(self, input, nbest_size):
++- return _sentencepiece.SentencePieceProcessor_NBestEncodeAsPieces(self, input, nbest_size)
++-
++- def NBestEncodeAsIds(self, input, nbest_size):
++- return _sentencepiece.SentencePieceProcessor_NBestEncodeAsIds(self, input, nbest_size)
++-
++- def SampleEncodeAsPieces(self, input, nbest_size, alpha):
++- return _sentencepiece.SentencePieceProcessor_SampleEncodeAsPieces(self, input, nbest_size, alpha)
++-
++- def SampleEncodeAsIds(self, input, nbest_size, alpha):
++- return _sentencepiece.SentencePieceProcessor_SampleEncodeAsIds(self, input, nbest_size, alpha)
++-
++ def SampleEncodeAndScoreAsPieces(self, input, num_samples, theta, wor, include_best):
++ return _sentencepiece.SentencePieceProcessor_SampleEncodeAndScoreAsPieces(self, input, num_samples, theta, wor, include_best)
++
++ def SampleEncodeAndScoreAsIds(self, input, num_samples, theta, wor, include_best):
++ return _sentencepiece.SentencePieceProcessor_SampleEncodeAndScoreAsIds(self, input, num_samples, theta, wor, include_best)
++
++- def DecodePieces(self, pieces):
++- return _sentencepiece.SentencePieceProcessor_DecodePieces(self, pieces)
++-
++ def CalculateEntropy(self, text, theta):
++ return _sentencepiece.SentencePieceProcessor_CalculateEntropy(self, text, theta)
++
++- def EncodeAsSerializedProto(self, input):
++- return _sentencepiece.SentencePieceProcessor_EncodeAsSerializedProto(self, input)
++-
++- def SampleEncodeAsSerializedProto(self, input, nbest_size, alpha):
++- return _sentencepiece.SentencePieceProcessor_SampleEncodeAsSerializedProto(self, input, nbest_size, alpha)
++-
++- def NBestEncodeAsSerializedProto(self, input, nbest_size):
++- return _sentencepiece.SentencePieceProcessor_NBestEncodeAsSerializedProto(self, input, nbest_size)
++-
++- def DecodePiecesAsSerializedProto(self, pieces):
++- return _sentencepiece.SentencePieceProcessor_DecodePiecesAsSerializedProto(self, pieces)
++-
++ def GetPieceSize(self):
++ return _sentencepiece.SentencePieceProcessor_GetPieceSize(self)
++
++@@ -171,30 +138,69 @@ class SentencePieceProcessor(object):
++ def LoadFromFile(self, arg):
++ return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
++
++- def DecodeIdsWithCheck(self, ids):
++- return _sentencepiece.SentencePieceProcessor_DecodeIdsWithCheck(self, ids)
+++ def _EncodeAsIds(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsIds(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _EncodeAsPieces(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsPieces(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _EncodeAsSerializedProto(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsSerializedProto(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _EncodeAsIdsBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsIdsBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _EncodeAsPiecesBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsPiecesBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _EncodeAsSerializedProtoBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsSerializedProtoBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
++- def DecodeIdsAsSerializedProtoWithCheck(self, ids):
++- return _sentencepiece.SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck(self, ids)
+++ def _DecodeIds(self, ids):
+++ return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
++
++- def _EncodeAsIds(self, text, enabele_sampling, nbest_size, alpha, add_bos, add_eos, reverse):
++- return _sentencepiece.SentencePieceProcessor__EncodeAsIds(self, text, enabele_sampling, nbest_size, alpha, add_bos, add_eos, reverse)
+++ def _DecodePieces(self, pieces):
+++ return _sentencepiece.SentencePieceProcessor__DecodePieces(self, pieces)
++
++- def _EncodeAsPieces(self, text, enabele_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
++- return _sentencepiece.SentencePieceProcessor__EncodeAsPieces(self, text, enabele_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ def _DecodeIdsAsSerializedProto(self, ids):
+++ return _sentencepiece.SentencePieceProcessor__DecodeIdsAsSerializedProto(self, ids)
++
++- def _NBestEncodeAsIds(self, text, nbest_size, add_bos, add_eos, reverse):
++- return _sentencepiece.SentencePieceProcessor__NBestEncodeAsIds(self, text, nbest_size, add_bos, add_eos, reverse)
+++ def _DecodePiecesAsSerializedProto(self, pieces):
+++ return _sentencepiece.SentencePieceProcessor__DecodePiecesAsSerializedProto(self, pieces)
+++
+++ def _DecodeIdsBatch(self, ins, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__DecodeIdsBatch(self, ins, num_threads)
+++
+++ def _DecodeIdsAsSerializedProtoBatch(self, ins, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch(self, ins, num_threads)
+++
+++ def _DecodePiecesBatch(self, ins, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__DecodePiecesBatch(self, ins, num_threads)
+++
+++ def _DecodePiecesAsSerializedProtoBatch(self, ins, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(self, ins, num_threads)
+++
+++ def _NBestEncodeAsIds(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__NBestEncodeAsIds(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
++
++ def _NBestEncodeAsPieces(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__NBestEncodeAsPieces(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
++
++- def _SampleEncodeAndScoreAsIds(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse):
++- return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsIds(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse)
+++ def _NBestEncodeAsSerializedProto(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__NBestEncodeAsSerializedProto(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _SampleEncodeAndScoreAsIds(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsIds(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
++
++ def _SampleEncodeAndScoreAsPieces(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsPieces(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
++
+++ def _CalculateEntropy(self, text, theta):
+++ return _sentencepiece.SentencePieceProcessor__CalculateEntropy(self, text, theta)
+++
+++ def _CalculateEntropyBatch(self, ins, theta, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__CalculateEntropyBatch(self, ins, theta, num_threads)
+++
++ def Init(self,
++ model_file=None,
++ model_proto=None,
++@@ -205,7 +211,8 @@ class SentencePieceProcessor(object):
++ emit_unk_piece=False,
++ enable_sampling=False,
++ nbest_size=-1,
++- alpha=0.1):
+++ alpha=0.1,
+++ num_threads=1):
++ """Initialzie sentencepieceProcessor.
++
++ Args:
++@@ -225,6 +232,7 @@ class SentencePieceProcessor(object):
++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and dropout probability of
++ merge operations for BPE-dropout.
+++ num_threads: number of threads in batch processing.
++ """
++
++ _sentencepiece_processor_init_native(self)
++@@ -236,6 +244,7 @@ class SentencePieceProcessor(object):
++ self._enable_sampling = enable_sampling
++ self._nbest_size = nbest_size
++ self._alpha = alpha
+++ self._num_threads = num_threads
++ if model_file or model_proto:
++ self.Load(model_file=model_file, model_proto=model_proto)
++
++@@ -249,7 +258,8 @@ class SentencePieceProcessor(object):
++ emit_unk_piece=None,
++ enable_sampling=None,
++ nbest_size=None,
++- alpha=None):
+++ alpha=None,
+++ num_threads=None):
++ """Encode text input to segmented ids or tokens.
++
++ Args:
++@@ -268,6 +278,7 @@ class SentencePieceProcessor(object):
++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and merge probability for
++ BPE-dropout (probablity 'p' in BPE-dropout paper).
+++ num_threads: the number of threads used in the batch processin (Default = 1).
++ """
++
++ if out_type is None:
++@@ -286,6 +297,8 @@ class SentencePieceProcessor(object):
++ nbest_size = self._nbest_size
++ if alpha is None:
++ alpha = self._alpha
+++ if num_threads is None:
+++ num_threads = self._num_threads
++
++ if enable_sampling == True and (nbest_size is None or nbest_size == 0 or
++ nbest_size == 1 or alpha is None):
++@@ -296,18 +309,59 @@ class SentencePieceProcessor(object):
++ 'instead of nbest segmentations.'
++ )
++
++- def _encode(text):
++- if out_type is int:
++- return self._EncodeAsIds(text, enable_sampling, nbest_size,
++- alpha, add_bos, add_eos, reverse)
++- else:
++- return self._EncodeAsPieces(text, enable_sampling, nbest_size,
++- alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if num_threads is None or type(num_threads) is not int:
+++ raise RuntimeError('num_threads must be int')
++
++ if type(input) is list:
++- return [_encode(n) for n in input]
+++ if out_type is int:
+++ return self._EncodeAsIdsBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type is str:
+++ return self._EncodeAsPiecesBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'proto':
+++ return self._EncodeAsSerializedProtoBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ if out_type is int:
+++ return self._EncodeAsIds(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type is str:
+++ return self._EncodeAsPieces(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'proto':
+++ return self._EncodeAsSerializedProto(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ raise RuntimeError('unknown out_type={}'.format(out_type))
+++ return None
++
++- return _encode(input)
+++
+++ def EncodeAsPieces(self, input, **kwargs):
+++ return self.Encode(input=input, out_type=str, **kwargs)
+++
+++
+++ def EncodeAsIds(self, input, **kwargs):
+++ return self.Encode(input=input, out_type=int, **kwargs)
+++
+++
+++ def EncodeAsSerializedProto(self, input, **kwargs):
+++ return self.Encode(input=input, out_type='proto', **kwargs)
+++
+++
+++ def SampleEncodeAsPieces(self, input, nbest_size=None, alpha=None, **kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type=str, enable_sampling=True, **kwargs)
+++
+++
+++ def SampleEncodeAsIds(self, input, nbest_size=None, alpha=None,**kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type=int, enable_sampling=True, **kwargs)
+++
+++
+++ def SampleEncodeAsSerializedProto(self, input, nbest_size=None, alpha=None, **kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type='proto', enable_sampling=True, **kwargs)
++
++
++ def NBestEncode(self,
++@@ -348,9 +402,14 @@ class SentencePieceProcessor(object):
++
++ def _encode(text):
++ if out_type is int:
++- return self._NBestEncodeAsIds(text, nbest_size, add_bos, add_eos, reverse)
++- else:
++- return self._NBestEncodeAsPieces(text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
+++ return self._NBestEncodeAsIds(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type is str:
+++ return self._NBestEncodeAsPieces(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'proto':
+++ return self._NBestEncodeAsSerializedProto(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++@@ -358,6 +417,21 @@ class SentencePieceProcessor(object):
++ return _encode(input)
++
++
+++ def NBestEncodeAsPieces(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type=str, **kwargs)
+++
+++
+++ def NBestEncodeAsIds(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type=int, **kwargs)
+++
+++
+++ def NBestEncodeAsSerializedProto(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type='proto', **kwargs)
+++
+++
++ def SampleEncodeAndScore(self,
++ input,
++ out_type=None,
++@@ -373,7 +447,7 @@ class SentencePieceProcessor(object):
++
++ Args:
++ input: input string. accepsts list of string.
++- out_type: output type. int or str.
+++ out_type: output type. int or str or 'proto'.
++ add_bos: Add <s> to the result (Default = false)
++ add_eos: Add </s> to the result (Default = false) <s>/</s> is added after reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++@@ -413,7 +487,7 @@ class SentencePieceProcessor(object):
++ def _encode(text):
++ if out_type is int:
++ return self._SampleEncodeAndScoreAsIds(text, num_samples, theta, wor, include_best,
++- add_bos, add_eos, reverse)
+++ add_bos, add_eos, reverse, emit_unk_piece)
++ else:
++ return self._SampleEncodeAndScoreAsPieces(text, num_samples, theta, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++@@ -424,35 +498,90 @@ class SentencePieceProcessor(object):
++ return _encode(input)
++
++
++- def Decode(self, input):
++- """Decode processed id or token sequences."""
+++ def Decode(self, input, out_type=str, num_threads=None):
+++ """Decode processed id or token sequences.
+++
+++ Args:
+++ out_type: output type. str or 'proto' (Default = str)
+++ num_threads: the number of threads used in the batch processin (Default = 1).
+++ """
+++
+++ if num_threads is None:
+++ num_threads = self._num_threads
+++
+++ if num_threads is None or type(num_threads) is not int:
+++ raise RuntimeError('num_threads must be int')
++
++ if not input:
++- return self.DecodeIds([])
++- elif type(input) is int:
++- return self.DecodeIdsWithCheck([input])
++- elif type(input) is str:
++- return self.DecodePieces([input])
+++ return ''
+++
+++ if out_type is str:
+++ if type(input) is int:
+++ return self._DecodeIds([input])
+++ if type(input) is str:
+++ return self._DecodePieces([input])
+++
+++ if type(input) is list:
+++ if len(input) == 0 or type(input[0]) is int:
+++ return self._DecodeIds(input)
+++ if type(input[0]) is str:
+++ return self._DecodePieces(input)
+++
+++ if type(input[0]) is list:
+++ if len(input[0]) == 0 or type(input[0][0]) is int:
+++ return self._DecodeIdsBatch(input, num_threads)
+++ if type(input[0][0]) is str:
+++ return self._DecodePiecesBatch(input, num_threads)
+++
+++ if out_type == 'proto':
+++ if type(input) is int:
+++ return self._DecodeIdsAsSerializedProto([input])
+++ if type(input) is str:
+++ return self._DecodePiecesAsSerializedProto([input])
+++
+++ if type(input) is list:
+++ if len(input) == 0 or type(input[0]) is int:
+++ return self._DecodeIdsAsSerializedProto(input)
+++ if type(input[0]) is str:
+++ return self._DecodePiecesAsSerializedProto(input)
+++
+++ if type(input[0]) is list:
+++ if len(input[0]) == 0 or type(input[0][0]) is int:
+++ return self._DecodeIdsAsSerializedProtoBatch(input, num_threads)
+++ if type(input[0][0]) is str:
+++ return self._DecodePiecesAsSerializedProtoBatch(input, num_threads)
+++
+++
+++ raise RuntimeError('unknown output or input type')
+++ return None
++
++- def _decode(input):
++- if not input:
++- return self.DecodeIds([])
++- if type(input[0]) is int:
++- return self.DecodeIdsWithCheck(input)
++- return self.DecodePieces(input)
++
++- if type(input[0]) is list:
++- return [_decode(n) for n in input]
+++ def DecodePieces(self, input, out_type=str, **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++- return _decode(input)
++
+++ def DecodeIds(self, input, out_type=str, **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def DecodePiecesAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++- def Entropy(self, input, theta):
++- """Calculate sentence entropy"""
++
+++ def DecodeIdsAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def CalculateEntropy(self, input, theta, num_threads=None):
+++ """Calculate sentence entropy"""
++ if type(input) is list:
++- return [self.CalculateEntropy(n, theta) for n in input]
++- return self.CalculateEntropy(input, theta)
+++ if num_threads is None:
+++ num_threads = self._num_threads
+++ if num_threads is None or type(num_threads) is not int:
+++ raise RuntimeError('num_threads must be int')
+++ return self._CalculateEntropyBatch(input, theta, num_threads)
+++
+++ return self._CalculateEntropy(input, theta)
++
++
++ def piece_size(self):
++@@ -642,8 +771,6 @@ setattr(SentencePieceProcessor, '__init__', SentencePieceProcessor.Init)
++
++ SentencePieceProcessor.Tokenize = SentencePieceProcessor.Encode
++ SentencePieceProcessor.Detokenize = SentencePieceProcessor.Decode
++-SentencePieceProcessor.DecodeIds = SentencePieceProcessor.DecodeIdsWithCheck
++-SentencePieceProcessor.DecodeIdsAsSerializedProto = SentencePieceProcessor.DecodeIdsAsSerializedProtoWithCheck
++
++ for m in [
++ 'PieceToId', 'IdToPiece', 'GetScore', 'IsUnknown', 'IsControl', 'IsUnused',
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 21bb7cf..3a822bc 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -2,9 +2,13 @@
++ %include exception.i
++
++ %{
+++#include <iostream>
++ #include <algorithm>
+++#include <functional>
++ #include <limits>
++ #include <cmath>
+++#include <thread>
+++#include <vector>
++ #include <sentencepiece_processor.h>
++ #include <sentencepiece_trainer.h>
++
++@@ -12,6 +16,8 @@ namespace {
++ PyObject* kUnicodeInput = reinterpret_cast<PyObject* >(0x1);
++ PyObject* kByteInput = reinterpret_cast<PyObject* >(0x2);
++
+++using BytesArray = std::vector<sentencepiece::util::bytes>;
+++
++ inline void ReleaseResultObject(PyObject *obj) {
++ if (obj != nullptr && obj != kUnicodeInput && obj != kByteInput) {
++ Py_XDECREF(obj);
++@@ -54,7 +60,7 @@ PyObject* MakePyOutputString(const std::string& output,
++ return PyBytes_FromStringAndSize(output.data(), output.size());
++ }
++
++-PyObject* MakePyOutputBytes(const std::string& output) {
+++PyObject* MakePyOutputBytes(const sentencepiece::util::bytes& output) {
++ return PyBytes_FromStringAndSize(output.data(), output.size());
++ }
++
++@@ -126,18 +132,18 @@ class PySentenceIterator : public sentencepiece::SentenceIterator {
++ sentencepiece::util::Status status_;
++ };
++
++-void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
++- std::vector<int> *ids,
++- bool add_bos, bool add_eos, bool reverse) {
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ std::vector<int> *ids,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
++ if (!add_bos && !add_eos && !reverse) return;
++ if (reverse) std::reverse(ids->begin(), ids->end());
++ if (add_bos) ids->insert(ids->begin(), sp.bos_id());
++ if (add_eos) ids->push_back(sp.eos_id());
++ }
++
++-void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++- std::vector<std::string> *pieces,
++- bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ std::vector<std::string> *pieces,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
++ if (!add_bos && !add_eos && !reverse && !emit_unk_piece) return;
++ if (reverse) std::reverse(pieces->begin(), pieces->end());
++ if (add_bos) pieces->insert(pieces->begin(), sp.IdToPiece(sp.bos_id()));
++@@ -152,6 +158,98 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ }
++ }
++ }
+++
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ sentencepiece::util::bytes *proto,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
+++ if (add_bos || add_eos || reverse || emit_unk_piece) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kUnimplemented,
+++ "add_bos, add_eos, reverse, and emit_unk_piece is not supported in AsSerialize API");
+++ }
+++}
+++
+++inline void CheckIds(const std::vector<int> &ids, int num_pieces) {
+++ for (int id : ids) {
+++ if (id < 0 || id >= num_pieces) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kOutOfRange,
+++ "piece id is out of range.");
+++ }
+++ }
+++}
+++
+++inline void CheckIds(const std::vector<std::string> &ids, int num_pieces) {}
+++
+++class ThreadPool {
+++ public:
+++ explicit ThreadPool(size_t request_size) :
+++ request_size_(request_size) {}
+++
+++ virtual ~ThreadPool() {
+++ for (auto &task : tasks_) {
+++ task.join();
+++ }
+++ }
+++
+++ void Schedule(std::function<void()> closure) {
+++ static constexpr size_t kMinThreadSize = 2;
+++ if (request_size_ < kMinThreadSize) {
+++ closure();
+++ } else {
+++ tasks_.emplace_back(closure);
+++ }
+++ }
+++
+++ private:
+++ size_t request_size_ = 0;
+++ std::vector<std::thread> tasks_;
+++};
+++
+++template <typename T>
+++inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
+++ *num_threads = std::max<int>(1,
+++ std::min<int>({*num_threads,
+++ static_cast<int>(ins.size()), 256}));
+++}
+++
+++#define DEFINE_ENCODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
+++ std::vector<OutType> outs(ins.size()); \
+++ InitNumThreads(ins, &num_threads); \
+++ { \
+++ ThreadPool pool(ins.size()); \
+++ for (int n = 0; n < num_threads; ++n) { \
+++ pool.Schedule([&, n]() { \
+++ for (size_t i = n; i < ins.size(); i += num_threads) { \
+++ auto out = enable_sampling ? \
+++ self->Sample##FuncName(ins[i], \
+++ nbest_size, alpha) : \
+++ self->FuncName(ins[i]); \
+++ RewriteIds(*self, &out, add_bos, add_eos, reverse, \
+++ emit_unk_piece); \
+++ outs[i] = std::move(out); \
+++ } \
+++ }); \
+++ } \
+++ } \
+++ return outs;
+++
+++#define DEFINE_DECODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
+++ std::vector<OutType> outs(ins.size()); \
+++ InitNumThreads(ins, &num_threads); \
+++ { \
+++ ThreadPool pool(ins.size()); \
+++ for (int n = 0; n < num_threads; ++n) { \
+++ pool.Schedule([&, n]() { \
+++ for (size_t i = n; i < ins.size(); i += num_threads) { \
+++ CheckIds(ins[i], self->GetPieceSize()); \
+++ outs[i] = self->FuncName(ins[i]); \
+++ } \
+++ }); \
+++ } \
+++ } \
+++ return outs;
+++
++ } // namespace
++ %}
++
++@@ -171,15 +269,28 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ %ignore sentencepiece::SentencePieceText;
++ %ignore sentencepiece::NormalizerSpec;
++ %ignore sentencepiece::TrainerSpec;
++-
++ %ignore sentencepiece::SentencePieceProcessor::status;
+++
++ %ignore sentencepiece::SentencePieceProcessor::Encode;
+++%ignore sentencepiece::SentencePieceProcessor::EncodeAsPieces;
+++%ignore sentencepiece::SentencePieceProcessor::EncodeAsIds;
+++%ignore sentencepiece::SentencePieceProcessor::EncodeAsSerializedProto;
++ %ignore sentencepiece::SentencePieceProcessor::SampleEncode;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsIds;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsPieces;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsSerializedProto;
++ %ignore sentencepiece::SentencePieceProcessor::NBestEncode;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsPieces;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsIds;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsSerializedProto;
++ %ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScore;
+++
++ %ignore sentencepiece::SentencePieceProcessor::Decode;
++ %ignore sentencepiece::SentencePieceProcessor::DecodeIds;
+++%ignore sentencepiece::SentencePieceProcessor::DecodePieces;
+++%ignore sentencepiece::SentencePieceProcessor::DecodePiecesAsSerializedProto;
++ %ignore sentencepiece::SentencePieceProcessor::DecodeIdsAsSerializedProto;
+++
++ %ignore sentencepiece::SentencePieceProcessor::model_proto;
++ %ignore sentencepiece::SentencePieceProcessor::Load;
++ %ignore sentencepiece::SentencePieceProcessor::LoadOrDie;
++@@ -200,62 +311,131 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ return $self->Load(arg);
++ }
++
++- std::string DecodeIdsWithCheck(
++- const std::vector<int> &ids) const {
++- const int num_pieces = $self->GetPieceSize();
++- for (int id : ids) {
++- if (id < 0 || id >= num_pieces) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "piece id is out of range.");
++- }
++- }
++- return $self->DecodeIds(ids);
++- }
++-
++- util::bytes DecodeIdsAsSerializedProtoWithCheck(
++- const std::vector<int> &ids) const {
++- const int num_pieces = $self->GetPieceSize();
++- for (int id : ids) {
++- if (id < 0 || id >= num_pieces) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "piece id is out of range.");
++- }
++- }
++- return $self->DecodeIdsAsSerializedProto(ids);
++- }
++-
+++ /////////////////////////////////////////////////////////////////////////////
+++ // EncodeAs* (Single request)
++ std::vector<int> _EncodeAsIds(absl::string_view text,
++- bool enabele_sampling,
+++ bool enable_sampling,
++ int nbest_size, float alpha,
++- bool add_bos, bool add_eos, bool reverse) {
++- auto ids = enabele_sampling ?
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ auto ids = enable_sampling ?
++ $self->SampleEncodeAsIds(text, nbest_size, alpha) :
++ $self->EncodeAsIds(text);
++- RewriteIds(*$self, &ids, add_bos, add_eos, reverse);
+++ RewriteIds(*$self, &ids, add_bos, add_eos, reverse, emit_unk_piece);
++ return ids;
++ }
++
++ std::vector<std::string> _EncodeAsPieces(absl::string_view text,
++- bool enabele_sampling,
+++ bool enable_sampling,
++ int nbest_size, float alpha,
++ bool add_bos, bool add_eos, bool reverse,
++- bool emit_unk_piece) {
++- auto pieces = enabele_sampling ?
+++ bool emit_unk_piece) const {
+++ auto pieces = enable_sampling ?
++ $self->SampleEncodeAsPieces(text, nbest_size, alpha) :
++ $self->EncodeAsPieces(text);
++- RewritePieces(*$self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
+++ RewriteIds(*$self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
++ return pieces;
++ }
++
+++ sentencepiece::util::bytes _EncodeAsSerializedProto(absl::string_view text,
+++ bool enable_sampling,
+++ int nbest_size, float alpha,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ auto proto = enable_sampling ?
+++ $self->SampleEncodeAsSerializedProto(text, nbest_size, alpha) :
+++ $self->EncodeAsSerializedProto(text);
+++ RewriteIds(*$self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
+++ return proto;
+++ }
+++
+++ /////////////////////////////////////////////////////////////////////////////
+++ // EncodeAs* (Batch request)
+++ std::vector<std::vector<int>> _EncodeAsIdsBatch(
+++ const std::vector<absl::string_view> &ins, int num_threads,
+++ bool enable_sampling, int nbest_size, float alpha,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsIds,
+++ absl::string_view, std::vector<int>);
+++ }
+++
+++ std::vector<std::vector<std::string>> _EncodeAsPiecesBatch(
+++ const std::vector<absl::string_view> &ins, int num_threads,
+++ bool enable_sampling, int nbest_size, float alpha,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsPieces,
+++ absl::string_view, std::vector<std::string>);
+++ }
+++
+++ BytesArray _EncodeAsSerializedProtoBatch(
+++ const std::vector<absl::string_view> &ins, int num_threads,
+++ bool enable_sampling, int nbest_size, float alpha,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsSerializedProto,
+++ absl::string_view,
+++ sentencepiece::util::bytes);
+++ }
+++
+++ /////////////////////////////////////////////////////////////////////////////
+++ // DecodeAs* (Single request)
+++ std::string _DecodeIds(const std::vector<int> &ids) const {
+++ CheckIds(ids, $self->GetPieceSize());
+++ return $self->DecodeIds(ids);
+++ }
+++
+++ std::string _DecodePieces(const std::vector<std::string> &pieces) const {
+++ return $self->DecodePieces(pieces);
+++ }
+++
+++ sentencepiece::util::bytes _DecodeIdsAsSerializedProto(
+++ const std::vector<int> &ids) const {
+++ CheckIds(ids, $self->GetPieceSize());
+++ return $self->DecodeIdsAsSerializedProto(ids);
+++ }
+++
+++ sentencepiece::util::bytes _DecodePiecesAsSerializedProto(
+++ const std::vector<std::string> &pieces) const {
+++ CheckIds(pieces, $self->GetPieceSize());
+++ return $self->DecodePiecesAsSerializedProto(pieces);
+++ }
+++
+++ /////////////////////////////////////////////////////////////////////////////
+++ // DecodeAs* (Batch request)
+++ std::vector<std::string> _DecodeIdsBatch(
+++ const std::vector<std::vector<int>> &ins, int num_threads) const {
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIds, int, std::string);
+++ }
+++
+++ BytesArray _DecodeIdsAsSerializedProtoBatch(
+++ const std::vector<std::vector<int>> &ins, int num_threads) const {
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIdsAsSerializedProto, int,
+++ sentencepiece::util::bytes);
+++ }
+++
+++ std::vector<std::string> _DecodePiecesBatch(
+++ const std::vector<std::vector<std::string>> &ins, int num_threads) const {
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePieces, std::string, std::string);
+++ }
+++
+++ BytesArray _DecodePiecesAsSerializedProtoBatch(
+++ const std::vector<std::vector<std::string>> &ins, int num_threads) const {
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsSerializedProto, std::string,
+++ sentencepiece::util::bytes);
+++ }
+++
+++ ////////////////////////////////////////////////////////////////////////////
+++ // NBestEncodeAs* (Single request)
++ std::vector<std::vector<int>>
++ _NBestEncodeAsIds(absl::string_view text,
++ int nbest_size,
++- bool add_bos, bool add_eos, bool reverse) {
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
++ auto idss = $self->NBestEncodeAsIds(text, nbest_size);
++ for (auto &ids : idss) {
++- RewriteIds(*$self, &ids, add_bos, add_eos, reverse);
+++ RewriteIds(*$self, &ids, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return idss;
++ }
++@@ -264,40 +444,74 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ _NBestEncodeAsPieces(absl::string_view text,
++ int nbest_size,
++ bool add_bos, bool add_eos, bool reverse,
++- bool emit_unk_piece) {
+++ bool emit_unk_piece) const {
++ auto piecess = $self->NBestEncodeAsPieces(text, nbest_size);
++ for (auto &pieces : piecess) {
++- RewritePieces(*$self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
+++ RewriteIds(*$self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return piecess;
++ }
++
+++ sentencepiece::util::bytes _NBestEncodeAsSerializedProto(absl::string_view text,
+++ int nbest_size,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ RewriteIds(*$self, static_cast<sentencepiece::util::bytes *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return $self->NBestEncodeAsSerializedProto(text, nbest_size);
+++ }
+++
+++ /////////////////////////////////////////////////////////////////////////////
+++ // SampleEncodeAndScoreAs* (Single request)
++ std::vector<std::pair<std::vector<int>, float>>
++ _SampleEncodeAndScoreAsIds(absl::string_view text,
++ int num_samples, float theta, bool wor,
++ bool include_best,
++- bool add_bos, bool add_eos, bool reverse) {
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
++ auto idss = $self->SampleEncodeAndScoreAsIds(text, num_samples,
++ theta, wor, include_best);
++ for (auto &ids : idss) {
++- RewriteIds(*$self, &ids.first, add_bos, add_eos, reverse);
+++ RewriteIds(*$self, &ids.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return idss;
++ }
++
++- std::vector<std::pair<std::vector<std::string>, float>>
+++ std::vector<std::pair<std::vector<std::string>, float>>
++ _SampleEncodeAndScoreAsPieces(absl::string_view text,
++ int num_samples, float theta, bool wor,
++ bool include_best,
++ bool add_bos, bool add_eos, bool reverse,
++- bool emit_unk_piece) {
+++ bool emit_unk_piece) const {
++ auto piecess = $self->SampleEncodeAndScoreAsPieces(text, num_samples,
++ theta, wor, include_best);
++ for (auto &pieces : piecess) {
++- RewritePieces(*$self, &pieces.first, add_bos, add_eos, reverse, emit_unk_piece);
+++ RewriteIds(*$self, &pieces.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return piecess;
++- }
+++ }
+++
+++ // Calculate Entropy
+++ float _CalculateEntropy(absl::string_view text, float theta) {
+++ return $self->CalculateEntropy(text, theta);
+++ }
+++
+++ std::vector<float> _CalculateEntropyBatch(const std::vector<absl::string_view> &ins,
+++ float theta, int num_threads) {
+++ std::vector<float> outs(ins.size());
+++ InitNumThreads(ins, &num_threads);
+++ {
+++ ThreadPool pool(ins.size());
+++ for (int n = 0; n < num_threads; ++n) {
+++ pool.Schedule([&, n]() {
+++ for (size_t i = n; i < ins.size(); i += num_threads) {
+++ outs[i] = self->CalculateEntropy(ins[i], theta);
+++ }
+++ });
+++ }
+++ }
+++ return outs;
+++ }
++
++ %pythoncode {
++ def Init(self,
++@@ -310,7 +524,8 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ emit_unk_piece=False,
++ enable_sampling=False,
++ nbest_size=-1,
++- alpha=0.1):
+++ alpha=0.1,
+++ num_threads=1):
++ """Initialzie sentencepieceProcessor.
++
++ Args:
++@@ -330,6 +545,7 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and dropout probability of
++ merge operations for BPE-dropout.
+++ num_threads: number of threads in batch processing.
++ """
++
++ _sentencepiece_processor_init_native(self)
++@@ -341,6 +557,7 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ self._enable_sampling = enable_sampling
++ self._nbest_size = nbest_size
++ self._alpha = alpha
+++ self._num_threads = num_threads
++ if model_file or model_proto:
++ self.Load(model_file=model_file, model_proto=model_proto)
++
++@@ -354,7 +571,8 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ emit_unk_piece=None,
++ enable_sampling=None,
++ nbest_size=None,
++- alpha=None):
+++ alpha=None,
+++ num_threads=None):
++ """Encode text input to segmented ids or tokens.
++
++ Args:
++@@ -373,6 +591,7 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and merge probability for
++ BPE-dropout (probablity 'p' in BPE-dropout paper).
+++ num_threads: the number of threads used in the batch processin (Default = 1).
++ """
++
++ if out_type is None:
++@@ -391,6 +610,8 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ nbest_size = self._nbest_size
++ if alpha is None:
++ alpha = self._alpha
+++ if num_threads is None:
+++ num_threads = self._num_threads
++
++ if enable_sampling == True and (nbest_size is None or nbest_size == 0 or
++ nbest_size == 1 or alpha is None):
++@@ -401,18 +622,59 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ 'instead of nbest segmentations.'
++ )
++
++- def _encode(text):
++- if out_type is int:
++- return self._EncodeAsIds(text, enable_sampling, nbest_size,
++- alpha, add_bos, add_eos, reverse)
++- else:
++- return self._EncodeAsPieces(text, enable_sampling, nbest_size,
++- alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if num_threads is None or type(num_threads) is not int:
+++ raise RuntimeError('num_threads must be int')
++
++ if type(input) is list:
++- return [_encode(n) for n in input]
+++ if out_type is int:
+++ return self._EncodeAsIdsBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type is str:
+++ return self._EncodeAsPiecesBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'proto':
+++ return self._EncodeAsSerializedProtoBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ if out_type is int:
+++ return self._EncodeAsIds(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type is str:
+++ return self._EncodeAsPieces(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'proto':
+++ return self._EncodeAsSerializedProto(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ raise RuntimeError('unknown out_type={}'.format(out_type))
+++ return None
++
++- return _encode(input)
+++
+++ def EncodeAsPieces(self, input, **kwargs):
+++ return self.Encode(input=input, out_type=str, **kwargs)
+++
+++
+++ def EncodeAsIds(self, input, **kwargs):
+++ return self.Encode(input=input, out_type=int, **kwargs)
+++
+++
+++ def EncodeAsSerializedProto(self, input, **kwargs):
+++ return self.Encode(input=input, out_type='proto', **kwargs)
+++
+++
+++ def SampleEncodeAsPieces(self, input, nbest_size=None, alpha=None, **kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type=str, enable_sampling=True, **kwargs)
+++
+++
+++ def SampleEncodeAsIds(self, input, nbest_size=None, alpha=None,**kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type=int, enable_sampling=True, **kwargs)
+++
+++
+++ def SampleEncodeAsSerializedProto(self, input, nbest_size=None, alpha=None, **kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type='proto', enable_sampling=True, **kwargs)
++
++
++ def NBestEncode(self,
++@@ -453,9 +715,14 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++
++ def _encode(text):
++ if out_type is int:
++- return self._NBestEncodeAsIds(text, nbest_size, add_bos, add_eos, reverse)
++- else:
++- return self._NBestEncodeAsPieces(text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
+++ return self._NBestEncodeAsIds(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type is str:
+++ return self._NBestEncodeAsPieces(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'proto':
+++ return self._NBestEncodeAsSerializedProto(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++@@ -463,6 +730,21 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ return _encode(input)
++
++
+++ def NBestEncodeAsPieces(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type=str, **kwargs)
+++
+++
+++ def NBestEncodeAsIds(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type=int, **kwargs)
+++
+++
+++ def NBestEncodeAsSerializedProto(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type='proto', **kwargs)
+++
+++
++ def SampleEncodeAndScore(self,
++ input,
++ out_type=None,
++@@ -478,7 +760,7 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++
++ Args:
++ input: input string. accepsts list of string.
++- out_type: output type. int or str.
+++ out_type: output type. int or str or 'proto'.
++ add_bos: Add <s> to the result (Default = false)
++ add_eos: Add </s> to the result (Default = false) <s>/</s> is added after reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++@@ -513,12 +795,12 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++
++ if include_best and not wor:
++ raise RuntimeError('When include_best is True, We must specify "wor = True".')
++-
+++
++
++ def _encode(text):
++ if out_type is int:
++ return self._SampleEncodeAndScoreAsIds(text, num_samples, theta, wor, include_best,
++- add_bos, add_eos, reverse)
+++ add_bos, add_eos, reverse, emit_unk_piece)
++ else:
++ return self._SampleEncodeAndScoreAsPieces(text, num_samples, theta, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++@@ -529,35 +811,90 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ return _encode(input)
++
++
++- def Decode(self, input):
++- """Decode processed id or token sequences."""
+++ def Decode(self, input, out_type=str, num_threads=None):
+++ """Decode processed id or token sequences.
+++
+++ Args:
+++ out_type: output type. str or 'proto' (Default = str)
+++ num_threads: the number of threads used in the batch processin (Default = 1).
+++ """
+++
+++ if num_threads is None:
+++ num_threads = self._num_threads
+++
+++ if num_threads is None or type(num_threads) is not int:
+++ raise RuntimeError('num_threads must be int')
++
++ if not input:
++- return self.DecodeIds([])
++- elif type(input) is int:
++- return self.DecodeIdsWithCheck([input])
++- elif type(input) is str:
++- return self.DecodePieces([input])
+++ return ''
+++
+++ if out_type is str:
+++ if type(input) is int:
+++ return self._DecodeIds([input])
+++ if type(input) is str:
+++ return self._DecodePieces([input])
+++
+++ if type(input) is list:
+++ if len(input) == 0 or type(input[0]) is int:
+++ return self._DecodeIds(input)
+++ if type(input[0]) is str:
+++ return self._DecodePieces(input)
+++
+++ if type(input[0]) is list:
+++ if len(input[0]) == 0 or type(input[0][0]) is int:
+++ return self._DecodeIdsBatch(input, num_threads)
+++ if type(input[0][0]) is str:
+++ return self._DecodePiecesBatch(input, num_threads)
+++
+++ if out_type == 'proto':
+++ if type(input) is int:
+++ return self._DecodeIdsAsSerializedProto([input])
+++ if type(input) is str:
+++ return self._DecodePiecesAsSerializedProto([input])
+++
+++ if type(input) is list:
+++ if len(input) == 0 or type(input[0]) is int:
+++ return self._DecodeIdsAsSerializedProto(input)
+++ if type(input[0]) is str:
+++ return self._DecodePiecesAsSerializedProto(input)
+++
+++ if type(input[0]) is list:
+++ if len(input[0]) == 0 or type(input[0][0]) is int:
+++ return self._DecodeIdsAsSerializedProtoBatch(input, num_threads)
+++ if type(input[0][0]) is str:
+++ return self._DecodePiecesAsSerializedProtoBatch(input, num_threads)
+++
+++
+++ raise RuntimeError('unknown output or input type')
+++ return None
++
++- def _decode(input):
++- if not input:
++- return self.DecodeIds([])
++- if type(input[0]) is int:
++- return self.DecodeIdsWithCheck(input)
++- return self.DecodePieces(input)
++
++- if type(input[0]) is list:
++- return [_decode(n) for n in input]
+++ def DecodePieces(self, input, out_type=str, **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++- return _decode(input)
++
+++ def DecodeIds(self, input, out_type=str, **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def DecodePiecesAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++- def Entropy(self, input, theta):
++- """Calculate sentence entropy"""
++
+++ def DecodeIdsAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def CalculateEntropy(self, input, theta, num_threads=None):
+++ """Calculate sentence entropy"""
++ if type(input) is list:
++- return [self.CalculateEntropy(n, theta) for n in input]
++- return self.CalculateEntropy(input, theta)
+++ if num_threads is None:
+++ num_threads = self._num_threads
+++ if num_threads is None or type(num_threads) is not int:
+++ raise RuntimeError('num_threads must be int')
+++ return self._CalculateEntropyBatch(input, theta, num_threads)
+++
+++ return self._CalculateEntropy(input, theta)
++
++
++ def piece_size(self):
++@@ -696,6 +1033,13 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ }
++ }
++
+++%typemap(out) std::vector<float> {
+++ $result = PyList_New($1.size());
+++ for (size_t i = 0; i < $1.size(); ++i) {
+++ PyList_SetItem($result, i, PyFloat_FromDouble(static_cast<double>($1[i])));
+++ }
+++}
+++
++ %typemap(out) std::vector<std::vector<int>> {
++ $result = PyList_New($1.size());
++ for (size_t i = 0; i < $1.size(); ++i) {
++@@ -715,6 +1059,13 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ }
++ }
++
+++%typemap(out) BytesArray {
+++ $result = PyList_New($1.size());
+++ for (size_t i = 0; i < $1.size(); ++i) {
+++ PyList_SetItem($result, i, MakePyOutputBytes($1[i]));
+++ }
+++}
+++
++ %typemap(out) std::vector<std::vector<std::string>> {
++ PyObject *input_type = resultobj;
++ $result = PyList_New($1.size());
++@@ -778,7 +1129,51 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem($input, i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = std::string(ustring.data(), ustring.size());
+++ (*out)[i].assign(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ $1 = out;
+++}
+++
+++%typemap(in) const std::vector<absl::string_view>& {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check($input)) {
+++ const size_t size = PyList_Size($input);
+++ out = new std::vector<std::string>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem($input, i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ $1 = out;
+++}
+++
+++%typemap(in) const std::vector<absl::string_view>& {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check($input)) {
+++ const size_t size = PyList_Size($input);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem($input, i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = absl::string_view(ustring.data(), ustring.size());
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -813,6 +1208,69 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ $1 = out;
++ }
++
+++%typemap(in) const std::vector<std::vector<std::string>>& {
+++ std::vector<std::vector<std::string>> *out = nullptr;
+++ if (PyList_Check($input)) {
+++ const size_t size = PyList_Size($input);
+++ out = new std::vector<std::vector<std::string>>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem($input, i);
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ const PyInputString ustring(PyList_GetItem(o, j));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i][j].assign(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ $1 = out;
+++}
+++
+++%typemap(in) const std::vector<std::vector<int>>& {
+++ std::vector<std::vector<int>> *out = nullptr;
+++ if (PyList_Check($input)) {
+++ const size_t size = PyList_Size($input);
+++ out = new std::vector<std::vector<int>>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem($input, i);
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ PyObject *o2 = PyList_GetItem(o, j);
+++ if (PyInt_Check(o2)) {
+++ (*out)[i][j] = static_cast<int>(PyInt_AsLong(o2));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ $1 = out;
+++}
+++
++ %typemap(in) const std::unordered_map<std::string, std::string> & {
++ std::unordered_map<std::string, std::string> *out = nullptr;
++ if (PyDict_Check($input)) {
++@@ -880,6 +1338,10 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ delete $1;
++ }
++
+++%typemap(freearg) const std::vector<absl::string_view>& {
+++ delete $1;
+++}
+++
++ %typemap(freearg) const std::vector<std::vector<std::string>>& {
++ delete $1;
++ }
++@@ -888,6 +1350,10 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ delete $1;
++ }
++
+++%typemap(freearg) const std::vector<float>& {
+++ delete $1;
+++}
+++
++ %typemap(freearg) const std::vector<std::vector<int>>& {
++ delete $1;
++ }
++@@ -948,8 +1414,6 @@ setattr(SentencePieceProcessor, '__init__', SentencePieceProcessor.Init)
++
++ SentencePieceProcessor.Tokenize = SentencePieceProcessor.Encode
++ SentencePieceProcessor.Detokenize = SentencePieceProcessor.Decode
++-SentencePieceProcessor.DecodeIds = SentencePieceProcessor.DecodeIdsWithCheck
++-SentencePieceProcessor.DecodeIdsAsSerializedProto = SentencePieceProcessor.DecodeIdsAsSerializedProtoWithCheck
++
++ for m in [
++ 'PieceToId', 'IdToPiece', 'GetScore', 'IsUnknown', 'IsControl', 'IsUnused',
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 36b3a0e..6df3880 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -2698,10 +2698,13 @@ SWIGINTERN PyObject *SWIG_PyStaticMethod_New(PyObject *SWIGUNUSEDPARM(self), PyO
++ #define SWIGTYPE_p_sentencepiece__SentencePieceTrainer swig_types[3]
++ #define SWIGTYPE_p_std__string swig_types[4]
++ #define SWIGTYPE_p_std__unordered_mapT_std__string_std__string_t swig_types[5]
++-#define SWIGTYPE_p_std__vectorT_int_t swig_types[6]
++-#define SWIGTYPE_p_std__vectorT_std__string_t swig_types[7]
++-static swig_type_info *swig_types[9];
++-static swig_module_info swig_module = {swig_types, 8, 0, 0, 0, 0};
+++#define SWIGTYPE_p_std__vectorT_absl__string_view_t swig_types[6]
+++#define SWIGTYPE_p_std__vectorT_int_t swig_types[7]
+++#define SWIGTYPE_p_std__vectorT_std__string_t swig_types[8]
+++#define SWIGTYPE_p_std__vectorT_std__vectorT_int_t_t swig_types[9]
+++#define SWIGTYPE_p_std__vectorT_std__vectorT_std__string_t_t swig_types[10]
+++static swig_type_info *swig_types[12];
+++static swig_module_info swig_module = {swig_types, 11, 0, 0, 0, 0};
++ #define SWIG_TypeQuery(name) SWIG_TypeQueryModule(&swig_module, &swig_module, name)
++ #define SWIG_MangledTypeQuery(name) SWIG_MangledTypeQueryModule(&swig_module, &swig_module, name)
++
++@@ -2805,9 +2808,13 @@ namespace swig {
++ }
++
++
+++#include <iostream>
++ #include <algorithm>
+++#include <functional>
++ #include <limits>
++ #include <cmath>
+++#include <thread>
+++#include <vector>
++ #include <sentencepiece_processor.h>
++ #include <sentencepiece_trainer.h>
++
++@@ -2815,6 +2822,8 @@ namespace {
++ PyObject* kUnicodeInput = reinterpret_cast<PyObject* >(0x1);
++ PyObject* kByteInput = reinterpret_cast<PyObject* >(0x2);
++
+++using BytesArray = std::vector<sentencepiece::util::bytes>;
+++
++ inline void ReleaseResultObject(PyObject *obj) {
++ if (obj != nullptr && obj != kUnicodeInput && obj != kByteInput) {
++ Py_XDECREF(obj);
++@@ -2857,7 +2866,7 @@ PyObject* MakePyOutputString(const std::string& output,
++ return PyBytes_FromStringAndSize(output.data(), output.size());
++ }
++
++-PyObject* MakePyOutputBytes(const std::string& output) {
+++PyObject* MakePyOutputBytes(const sentencepiece::util::bytes& output) {
++ return PyBytes_FromStringAndSize(output.data(), output.size());
++ }
++
++@@ -2929,18 +2938,18 @@ class PySentenceIterator : public sentencepiece::SentenceIterator {
++ sentencepiece::util::Status status_;
++ };
++
++-void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
++- std::vector<int> *ids,
++- bool add_bos, bool add_eos, bool reverse) {
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ std::vector<int> *ids,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
++ if (!add_bos && !add_eos && !reverse) return;
++ if (reverse) std::reverse(ids->begin(), ids->end());
++ if (add_bos) ids->insert(ids->begin(), sp.bos_id());
++ if (add_eos) ids->push_back(sp.eos_id());
++ }
++
++-void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++- std::vector<std::string> *pieces,
++- bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ std::vector<std::string> *pieces,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
++ if (!add_bos && !add_eos && !reverse && !emit_unk_piece) return;
++ if (reverse) std::reverse(pieces->begin(), pieces->end());
++ if (add_bos) pieces->insert(pieces->begin(), sp.IdToPiece(sp.bos_id()));
++@@ -2955,6 +2964,98 @@ void RewritePieces(const sentencepiece::SentencePieceProcessor &sp,
++ }
++ }
++ }
+++
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ sentencepiece::util::bytes *proto,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
+++ if (add_bos || add_eos || reverse || emit_unk_piece) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kUnimplemented,
+++ "add_bos, add_eos, reverse, and emit_unk_piece is not supported in AsSerialize API");
+++ }
+++}
+++
+++inline void CheckIds(const std::vector<int> &ids, int num_pieces) {
+++ for (int id : ids) {
+++ if (id < 0 || id >= num_pieces) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kOutOfRange,
+++ "piece id is out of range.");
+++ }
+++ }
+++}
+++
+++inline void CheckIds(const std::vector<std::string> &ids, int num_pieces) {}
+++
+++class ThreadPool {
+++ public:
+++ explicit ThreadPool(size_t request_size) :
+++ request_size_(request_size) {}
+++
+++ virtual ~ThreadPool() {
+++ for (auto &task : tasks_) {
+++ task.join();
+++ }
+++ }
+++
+++ void Schedule(std::function<void()> closure) {
+++ static constexpr size_t kMinThreadSize = 2;
+++ if (request_size_ < kMinThreadSize) {
+++ closure();
+++ } else {
+++ tasks_.emplace_back(closure);
+++ }
+++ }
+++
+++ private:
+++ size_t request_size_ = 0;
+++ std::vector<std::thread> tasks_;
+++};
+++
+++template <typename T>
+++inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
+++ *num_threads = std::max<int>(1,
+++ std::min<int>({*num_threads,
+++ static_cast<int>(ins.size()), 256}));
+++}
+++
+++#define DEFINE_ENCODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
+++ std::vector<OutType> outs(ins.size()); \
+++ InitNumThreads(ins, &num_threads); \
+++ { \
+++ ThreadPool pool(ins.size()); \
+++ for (int n = 0; n < num_threads; ++n) { \
+++ pool.Schedule([&, n]() { \
+++ for (size_t i = n; i < ins.size(); i += num_threads) { \
+++ auto out = enable_sampling ? \
+++ self->Sample##FuncName(ins[i], \
+++ nbest_size, alpha) : \
+++ self->FuncName(ins[i]); \
+++ RewriteIds(*self, &out, add_bos, add_eos, reverse, \
+++ emit_unk_piece); \
+++ outs[i] = std::move(out); \
+++ } \
+++ }); \
+++ } \
+++ } \
+++ return outs;
+++
+++#define DEFINE_DECODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
+++ std::vector<OutType> outs(ins.size()); \
+++ InitNumThreads(ins, &num_threads); \
+++ { \
+++ ThreadPool pool(ins.size()); \
+++ for (int n = 0; n < num_threads; ++n) { \
+++ pool.Schedule([&, n]() { \
+++ for (size_t i = n; i < ins.size(); i += num_threads) { \
+++ CheckIds(ins[i], self->GetPieceSize()); \
+++ outs[i] = self->FuncName(ins[i]); \
+++ } \
+++ }); \
+++ } \
+++ } \
+++ return outs;
+++
++ } // namespace
++
++
++@@ -3334,72 +3435,122 @@ SWIGINTERNINLINE PyObject*
++ SWIGINTERN sentencepiece::util::Status sentencepiece_SentencePieceProcessor_LoadFromFile(sentencepiece::SentencePieceProcessor *self,absl::string_view arg){
++ return self->Load(arg);
++ }
++-SWIGINTERN std::string sentencepiece_SentencePieceProcessor_DecodeIdsWithCheck(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
++- const int num_pieces = self->GetPieceSize();
++- for (int id : ids) {
++- if (id < 0 || id >= num_pieces) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "piece id is out of range.");
++- }
++- }
++- return self->DecodeIds(ids);
++- }
++-SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
++- const int num_pieces = self->GetPieceSize();
++- for (int id : ids) {
++- if (id < 0 || id >= num_pieces) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "piece id is out of range.");
++- }
++- }
++- return self->DecodeIdsAsSerializedProto(ids);
++- }
++-SWIGINTERN std::vector< int > sentencepiece_SentencePieceProcessor__EncodeAsIds(sentencepiece::SentencePieceProcessor *self,absl::string_view text,bool enabele_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse){
++- auto ids = enabele_sampling ?
+++SWIGINTERN std::vector< int > sentencepiece_SentencePieceProcessor__EncodeAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ auto ids = enable_sampling ?
++ self->SampleEncodeAsIds(text, nbest_size, alpha) :
++ self->EncodeAsIds(text);
++- RewriteIds(*self, &ids, add_bos, add_eos, reverse);
+++ RewriteIds(*self, &ids, add_bos, add_eos, reverse, emit_unk_piece);
++ return ids;
++ }
++-SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__EncodeAsPieces(sentencepiece::SentencePieceProcessor *self,absl::string_view text,bool enabele_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++- auto pieces = enabele_sampling ?
+++SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__EncodeAsPieces(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ auto pieces = enable_sampling ?
++ self->SampleEncodeAsPieces(text, nbest_size, alpha) :
++ self->EncodeAsPieces(text);
++- RewritePieces(*self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
+++ RewriteIds(*self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
++ return pieces;
++ }
++-SWIGINTERN std::vector< std::vector< int > > sentencepiece_SentencePieceProcessor__NBestEncodeAsIds(sentencepiece::SentencePieceProcessor *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse){
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__EncodeAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ auto proto = enable_sampling ?
+++ self->SampleEncodeAsSerializedProto(text, nbest_size, alpha) :
+++ self->EncodeAsSerializedProto(text);
+++ RewriteIds(*self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
+++ return proto;
+++ }
+++SWIGINTERN std::vector< std::vector< int > > sentencepiece_SentencePieceProcessor__EncodeAsIdsBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &ins,int num_threads,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsIds,
+++ absl::string_view, std::vector<int>);
+++ }
+++SWIGINTERN std::vector< std::vector< std::string > > sentencepiece_SentencePieceProcessor__EncodeAsPiecesBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &ins,int num_threads,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsPieces,
+++ absl::string_view, std::vector<std::string>);
+++ }
+++SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__EncodeAsSerializedProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &ins,int num_threads,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsSerializedProto,
+++ absl::string_view,
+++ sentencepiece::util::bytes);
+++ }
+++SWIGINTERN std::string sentencepiece_SentencePieceProcessor__DecodeIds(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
+++ CheckIds(ids, self->GetPieceSize());
+++ return self->DecodeIds(ids);
+++ }
+++SWIGINTERN std::string sentencepiece_SentencePieceProcessor__DecodePieces(sentencepiece::SentencePieceProcessor const *self,std::vector< std::string > const &pieces){
+++ return self->DecodePieces(pieces);
+++ }
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__DecodeIdsAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
+++ CheckIds(ids, self->GetPieceSize());
+++ return self->DecodeIdsAsSerializedProto(ids);
+++ }
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,std::vector< std::string > const &pieces){
+++ CheckIds(pieces, self->GetPieceSize());
+++ return self->DecodePiecesAsSerializedProto(pieces);
+++ }
+++SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodeIdsBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< int > > const &ins,int num_threads){
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIds, int, std::string);
+++ }
+++SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< int > > const &ins,int num_threads){
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIdsAsSerializedProto, int,
+++ sentencepiece::util::bytes);
+++ }
+++SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodePiecesBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< std::string > > const &ins,int num_threads){
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePieces, std::string, std::string);
+++ }
+++SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< std::string > > const &ins,int num_threads){
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsSerializedProto, std::string,
+++ sentencepiece::util::bytes);
+++ }
+++SWIGINTERN std::vector< std::vector< int > > sentencepiece_SentencePieceProcessor__NBestEncodeAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto idss = self->NBestEncodeAsIds(text, nbest_size);
++ for (auto &ids : idss) {
++- RewriteIds(*self, &ids, add_bos, add_eos, reverse);
+++ RewriteIds(*self, &ids, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return idss;
++ }
++-SWIGINTERN std::vector< std::vector< std::string > > sentencepiece_SentencePieceProcessor__NBestEncodeAsPieces(sentencepiece::SentencePieceProcessor *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++SWIGINTERN std::vector< std::vector< std::string > > sentencepiece_SentencePieceProcessor__NBestEncodeAsPieces(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto piecess = self->NBestEncodeAsPieces(text, nbest_size);
++ for (auto &pieces : piecess) {
++- RewritePieces(*self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
+++ RewriteIds(*self, &pieces, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return piecess;
++ }
++-SWIGINTERN std::vector< std::pair< std::vector< int >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds(sentencepiece::SentencePieceProcessor *self,absl::string_view text,int num_samples,float theta,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse){
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__NBestEncodeAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ RewriteIds(*self, static_cast<sentencepiece::util::bytes *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return self->NBestEncodeAsSerializedProto(text, nbest_size);
+++ }
+++SWIGINTERN std::vector< std::pair< std::vector< int >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float theta,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto idss = self->SampleEncodeAndScoreAsIds(text, num_samples,
++ theta, wor, include_best);
++ for (auto &ids : idss) {
++- RewriteIds(*self, &ids.first, add_bos, add_eos, reverse);
+++ RewriteIds(*self, &ids.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return idss;
++ }
++-SWIGINTERN std::vector< std::pair< std::vector< std::string >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(sentencepiece::SentencePieceProcessor *self,absl::string_view text,int num_samples,float theta,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++SWIGINTERN std::vector< std::pair< std::vector< std::string >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float theta,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto piecess = self->SampleEncodeAndScoreAsPieces(text, num_samples,
++ theta, wor, include_best);
++ for (auto &pieces : piecess) {
++- RewritePieces(*self, &pieces.first, add_bos, add_eos, reverse, emit_unk_piece);
+++ RewriteIds(*self, &pieces.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return piecess;
++ }
+++SWIGINTERN float sentencepiece_SentencePieceProcessor__CalculateEntropy(sentencepiece::SentencePieceProcessor *self,absl::string_view text,float theta){
+++ return self->CalculateEntropy(text, theta);
+++ }
+++SWIGINTERN std::vector< float > sentencepiece_SentencePieceProcessor__CalculateEntropyBatch(sentencepiece::SentencePieceProcessor *self,std::vector< absl::string_view > const &ins,float theta,int num_threads){
+++ std::vector<float> outs(ins.size());
+++ InitNumThreads(ins, &num_threads);
+++ {
+++ ThreadPool pool(ins.size());
+++ for (int n = 0; n < num_threads; ++n) {
+++ pool.Schedule([&, n]() {
+++ for (size_t i = n; i < ins.size(); i += num_threads) {
+++ outs[i] = self->CalculateEntropy(ins[i], theta);
+++ }
+++ });
+++ }
+++ }
+++ return outs;
+++ }
++
++ SWIGINTERN int
++ SWIG_AsVal_unsigned_SS_long (PyObject *obj, unsigned long *val)
++@@ -3703,7 +3854,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SetVocabulary(PyObject *SWIGUN
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = std::string(ustring.data(), ustring.size());
+++ (*out)[i].assign(ustring.data(), ustring.size());
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -3832,19 +3983,31 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
+++ int arg3 ;
+++ float arg4 ;
+++ bool arg5 ;
+++ bool arg6 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- std::vector< std::string > result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ float val4 ;
+++ int ecode4 = 0 ;
+++ bool val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ PyObject *swig_obj[6] ;
+++ std::vector< std::pair< std::vector< std::string >,float > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_EncodeAsPieces", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAndScoreAsPieces", 6, 6, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_EncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -3856,9 +4019,29 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsPieces(PyObject *SWIGU
++ resultobj = ustring.input_type();
++ arg2 = absl::string_view(ustring.data(), ustring.size());
++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "4"" of type '" "float""'");
+++ }
+++ arg4 = static_cast< float >(val4);
+++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "5"" of type '" "bool""'");
+++ }
+++ arg5 = static_cast< bool >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->EncodeAsPieces(arg2);
+++ result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAndScoreAsPieces(arg2,arg3,arg4,arg5,arg6);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -3869,7 +4052,11 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsPieces(PyObject *SWIGU
++ PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ PyObject *obj = PyList_New(result[i].first.size());
+++ for (size_t j = 0; j < result[i].first.size(); ++j) {
+++ PyList_SetItem(obj, j, MakePyOutputString(result[i].first[j], input_type));
+++ }
+++ PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++ }
++ }
++ return resultobj;
++@@ -3878,19 +4065,31 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
+++ int arg3 ;
+++ float arg4 ;
+++ bool arg5 ;
+++ bool arg6 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- std::vector< int > result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ float val4 ;
+++ int ecode4 = 0 ;
+++ bool val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ PyObject *swig_obj[6] ;
+++ std::vector< std::pair< std::vector< int >,float > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_EncodeAsIds", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAndScoreAsIds", 6, 6, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_EncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -3902,9 +4101,29 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsIds(PyObject *SWIGUNUS
++ resultobj = ustring.input_type();
++ arg2 = absl::string_view(ustring.data(), ustring.size());
++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "4"" of type '" "float""'");
+++ }
+++ arg4 = static_cast< float >(val4);
+++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "5"" of type '" "bool""'");
+++ }
+++ arg5 = static_cast< bool >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->EncodeAsIds(arg2);
+++ result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAndScoreAsIds(arg2,arg3,arg4,arg5,arg6);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -3914,7 +4133,11 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsIds(PyObject *SWIGUNUS
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, PyInt_FromLong(static_cast<long>(result[i])));
+++ PyObject *obj = PyList_New(result[i].first.size());
+++ for (size_t j = 0; j < result[i].first.size(); ++j) {
+++ PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
+++ }
+++ PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++ }
++ }
++ return resultobj;
++@@ -3923,22 +4146,22 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_NBestEncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
++- int arg3 ;
+++ float arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
+++ float val3 ;
++ int ecode3 = 0 ;
++ PyObject *swig_obj[3] ;
++- std::vector< std::vector< std::string > > result;
+++ float result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_NBestEncodeAsPieces", 3, 3, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_CalculateEntropy", 3, 3, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_NBestEncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -3950,113 +4173,71 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_NBestEncodeAsPieces(PyObject *
++ resultobj = ustring.input_type();
++ arg2 = absl::string_view(ustring.data(), ustring.size());
++ }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_NBestEncodeAsPieces" "', argument " "3"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "3"" of type '" "float""'");
++ }
++- arg3 = static_cast< int >(val3);
+++ arg3 = static_cast< float >(val3);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->NBestEncodeAsPieces(arg2,arg3);
+++ result = (float)((sentencepiece::SentencePieceProcessor const *)arg1)->CalculateEntropy(arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- PyObject *input_type = resultobj;
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].size());
++- for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i][j], input_type));
++- }
++- PyList_SetItem(resultobj, i, obj);
++- }
++- }
+++ resultobj = SWIG_From_float(static_cast< float >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_NBestEncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_GetPieceSize(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- PyObject *swig_obj[3] ;
++- std::vector< std::vector< int > > result;
+++ PyObject *swig_obj[1] ;
+++ int result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_NBestEncodeAsIds", 3, 3, swig_obj)) SWIG_fail;
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_NBestEncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_GetPieceSize" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_NBestEncodeAsIds" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->NBestEncodeAsIds(arg2,arg3);
+++ result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->GetPieceSize();
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].size());
++- for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
++- }
++- PyList_SetItem(resultobj, i, obj);
++- }
++- }
+++ resultobj = SWIG_From_int(static_cast< int >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_PieceToId(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- PyObject *swig_obj[4] ;
++- std::vector< std::string > result;
+++ PyObject *swig_obj[2] ;
+++ int result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAsPieces", 4, 4, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_PieceToId", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_PieceToId" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -4068,81 +4249,47 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAsPieces(PyObject
++ resultobj = ustring.input_type();
++ arg2 = absl::string_view(ustring.data(), ustring.size());
++ }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAsPieces" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAsPieces" "', argument " "4"" of type '" "float""'");
++- }
++- arg4 = static_cast< float >(val4);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAsPieces(arg2,arg3,arg4);
+++ result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->PieceToId(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- PyObject *input_type = resultobj;
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
++- }
++- }
+++ resultobj = SWIG_From_int(static_cast< int >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IdToPiece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
+++ int arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- PyObject *swig_obj[4] ;
++- std::vector< int > result;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ std::string *result = 0 ;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAsIds", 4, 4, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IdToPiece", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IdToPiece" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAsIds" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAsIds" "', argument " "4"" of type '" "float""'");
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IdToPiece" "', argument " "2"" of type '" "int""'");
++ }
++- arg4 = static_cast< float >(val4);
+++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAsIds(arg2,arg3,arg4);
+++ result = (std::string *) &((sentencepiece::SentencePieceProcessor const *)arg1)->IdToPiece(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -4150,10 +4297,8 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAsIds(PyObject *SW
++ }
++ }
++ {
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, PyInt_FromLong(static_cast<long>(result[i])));
++- }
+++ PyObject *input_type = resultobj;
+++ resultobj = MakePyOutputString(*result, input_type);
++ }
++ return resultobj;
++ fail:
++@@ -4161,489 +4306,290 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_GetScore(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
++- bool arg5 ;
++- bool arg6 ;
+++ int arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- bool val5 ;
++- int ecode5 = 0 ;
++- bool val6 ;
++- int ecode6 = 0 ;
++- PyObject *swig_obj[6] ;
++- std::vector< std::pair< std::vector< std::string >,float > > result;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ float result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAndScoreAsPieces", 6, 6, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_GetScore", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_GetScore" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "4"" of type '" "float""'");
++- }
++- arg4 = static_cast< float >(val4);
++- ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++- if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "5"" of type '" "bool""'");
++- }
++- arg5 = static_cast< bool >(val5);
++- ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++- if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "6"" of type '" "bool""'");
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_GetScore" "', argument " "2"" of type '" "int""'");
++ }
++- arg6 = static_cast< bool >(val6);
+++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAndScoreAsPieces(arg2,arg3,arg4,arg5,arg6);
+++ result = (float)((sentencepiece::SentencePieceProcessor const *)arg1)->GetScore(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- PyObject *input_type = resultobj;
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].first.size());
++- for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i].first[j], input_type));
++- }
++- PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++- }
++- }
+++ resultobj = SWIG_From_float(static_cast< float >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsUnknown(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
++- bool arg5 ;
++- bool arg6 ;
+++ int arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- bool val5 ;
++- int ecode5 = 0 ;
++- bool val6 ;
++- int ecode6 = 0 ;
++- PyObject *swig_obj[6] ;
++- std::vector< std::pair< std::vector< int >,float > > result;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ bool result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAndScoreAsIds", 6, 6, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsUnknown", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsUnknown" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "4"" of type '" "float""'");
++- }
++- arg4 = static_cast< float >(val4);
++- ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++- if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "5"" of type '" "bool""'");
++- }
++- arg5 = static_cast< bool >(val5);
++- ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++- if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "6"" of type '" "bool""'");
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsUnknown" "', argument " "2"" of type '" "int""'");
++ }
++- arg6 = static_cast< bool >(val6);
+++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAndScoreAsIds(arg2,arg3,arg4,arg5,arg6);
+++ result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsUnknown(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].first.size());
++- for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
++- }
++- PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++- }
++- }
+++ resultobj = SWIG_From_bool(static_cast< bool >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodePieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsControl(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::string > *arg2 = 0 ;
+++ int arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
+++ int val2 ;
+++ int ecode2 = 0 ;
++ PyObject *swig_obj[2] ;
++- std::string result;
+++ bool result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_DecodePieces", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsControl", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_DecodePieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsControl" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- std::vector<std::string> *out = nullptr;
++- if (PyList_Check(swig_obj[1])) {
++- const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::string>(size);
++- for (size_t i = 0; i < size; ++i) {
++- const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++- if (ustring.IsAvalable()) {
++- (*out)[i] = std::string(ustring.data(), ustring.size());
++- } else {
++- PyErr_SetString(PyExc_TypeError, "list must contain strings");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- }
++- } else {
++- PyErr_SetString(PyExc_TypeError, "not a list");
++- SWIG_fail;
++- }
++- arg2 = out;
++- }
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsControl" "', argument " "2"" of type '" "int""'");
+++ }
+++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->DecodePieces((std::vector< std::string > const &)*arg2);
+++ result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsControl(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- PyObject *input_type = resultobj;
++- resultobj = MakePyOutputString(result, input_type);
++- }
++- {
++- delete arg2;
++- }
+++ resultobj = SWIG_From_bool(static_cast< bool >(result));
++ return resultobj;
++ fail:
++- {
++- delete arg2;
++- }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsUnused(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- float arg3 ;
+++ int arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- float val3 ;
++- int ecode3 = 0 ;
++- PyObject *swig_obj[3] ;
++- float result;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ bool result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_CalculateEntropy", 3, 3, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsUnused", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsUnused" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "3"" of type '" "float""'");
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsUnused" "', argument " "2"" of type '" "int""'");
++ }
++- arg3 = static_cast< float >(val3);
+++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = (float)((sentencepiece::SentencePieceProcessor const *)arg1)->CalculateEntropy(arg2,arg3);
+++ result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsUnused(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_float(static_cast< float >(result));
+++ resultobj = SWIG_From_bool(static_cast< bool >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_EncodeAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsByte(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
+++ int arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
+++ int val2 ;
+++ int ecode2 = 0 ;
++ PyObject *swig_obj[2] ;
++- sentencepiece::util::bytes result;
+++ bool result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_EncodeAsSerializedProto", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsByte", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_EncodeAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsByte" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsByte" "', argument " "2"" of type '" "int""'");
+++ }
+++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->EncodeAsSerializedProto(arg2);
+++ result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsByte(arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- resultobj = MakePyOutputBytes(result);
++- }
+++ resultobj = SWIG_From_bool(static_cast< bool >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_unk_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- PyObject *swig_obj[4] ;
++- sentencepiece::util::bytes result;
+++ PyObject *swig_obj[1] ;
+++ int result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAsSerializedProto", 4, 4, swig_obj)) SWIG_fail;
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_unk_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAsSerializedProto" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAsSerializedProto" "', argument " "4"" of type '" "float""'");
++- }
++- arg4 = static_cast< float >(val4);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAsSerializedProto(arg2,arg3,arg4);
+++ result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->unk_id();
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- resultobj = MakePyOutputBytes(result);
++- }
+++ resultobj = SWIG_From_int(static_cast< int >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_NBestEncodeAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_bos_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- PyObject *swig_obj[3] ;
++- sentencepiece::util::bytes result;
+++ PyObject *swig_obj[1] ;
+++ int result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_NBestEncodeAsSerializedProto", 3, 3, swig_obj)) SWIG_fail;
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_NBestEncodeAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_bos_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_NBestEncodeAsSerializedProto" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->NBestEncodeAsSerializedProto(arg2,arg3);
+++ result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->bos_id();
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- resultobj = MakePyOutputBytes(result);
++- }
+++ resultobj = SWIG_From_int(static_cast< int >(result));
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodePiecesAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_eos_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::string > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- sentencepiece::util::bytes result;
+++ PyObject *swig_obj[1] ;
+++ int result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_DecodePiecesAsSerializedProto", 2, 2, swig_obj)) SWIG_fail;
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_DecodePiecesAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_eos_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- std::vector<std::string> *out = nullptr;
++- if (PyList_Check(swig_obj[1])) {
++- const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::string>(size);
++- for (size_t i = 0; i < size; ++i) {
++- const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++- if (ustring.IsAvalable()) {
++- (*out)[i] = std::string(ustring.data(), ustring.size());
++- } else {
++- PyErr_SetString(PyExc_TypeError, "list must contain strings");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- }
++- } else {
++- PyErr_SetString(PyExc_TypeError, "not a list");
++- SWIG_fail;
++- }
++- arg2 = out;
++- }
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->DecodePiecesAsSerializedProto((std::vector< std::string > const &)*arg2);
+++ result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->eos_id();
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- {
++- resultobj = MakePyOutputBytes(result);
++- }
++- {
++- delete arg2;
++- }
+++ resultobj = SWIG_From_int(static_cast< int >(result));
++ return resultobj;
++ fail:
++- {
++- delete arg2;
++- }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_GetPieceSize(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_pad_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ void *argp1 = 0 ;
++@@ -4655,12 +4601,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_GetPieceSize(PyObject *SWIGUNU
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_GetPieceSize" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_pad_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++ try {
++- result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->GetPieceSize();
+++ result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->pad_id();
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -4674,71 +4620,66 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_PieceToId(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_serialized_model_proto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- int result;
+++ PyObject *swig_obj[1] ;
+++ sentencepiece::util::bytes result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_PieceToId", 2, 2, swig_obj)) SWIG_fail;
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_PieceToId" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_serialized_model_proto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
++- }
++ {
++ try {
++- result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->PieceToId(arg2);
+++ result = ((sentencepiece::SentencePieceProcessor const *)arg1)->serialized_model_proto();
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_int(static_cast< int >(result));
+++ {
+++ resultobj = MakePyOutputBytes(result);
+++ }
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IdToPiece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_LoadFromFile(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- int arg2 ;
+++ absl::string_view arg2 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++ PyObject *swig_obj[2] ;
++- std::string *result = 0 ;
+++ sentencepiece::util::Status result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IdToPiece", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_LoadFromFile", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IdToPiece" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_LoadFromFile" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IdToPiece" "', argument " "2"" of type '" "int""'");
++- }
++- arg2 = static_cast< int >(val2);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = absl::string_view(ustring.data(), ustring.size());
+++ }
++ {
++ try {
++- result = (std::string *) &((sentencepiece::SentencePieceProcessor const *)arg1)->IdToPiece(arg2);
+++ result = sentencepiece_SentencePieceProcessor_LoadFromFile(arg1,arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -4746,8 +4687,10 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IdToPiece(PyObject *SWIGUNUSED
++ }
++ }
++ {
++- PyObject *input_type = resultobj;
++- resultobj = MakePyOutputString(*result, input_type);
+++ if (!(&result)->ok()) {
+++ SWIG_exception(ToSwigError((&result)->code()), (&result)->ToString().c_str());
+++ }
+++ resultobj = SWIG_From_bool((&result)->ok());
++ }
++ return resultobj;
++ fail:
++@@ -4755,338 +4698,916 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_GetScore(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- int arg2 ;
+++ absl::string_view arg2 ;
+++ bool arg3 ;
+++ int arg4 ;
+++ float arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- float result;
+++ bool val3 ;
+++ int ecode3 = 0 ;
+++ int val4 ;
+++ int ecode4 = 0 ;
+++ float val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ PyObject *swig_obj[9] ;
+++ std::vector< int > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_GetScore", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsIds", 9, 9, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_GetScore" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_GetScore" "', argument " "2"" of type '" "int""'");
++- }
++- arg2 = static_cast< int >(val2);
++ {
++- try {
++- result = (float)((sentencepiece::SentencePieceProcessor const *)arg1)->GetScore(arg2);
++- ReleaseResultObject(resultobj);
++- }
++- catch (const sentencepiece::util::Status &status) {
++- SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
++ }
+++ resultobj = ustring.input_type();
+++ arg2 = absl::string_view(ustring.data(), ustring.size());
++ }
++- resultobj = SWIG_From_float(static_cast< float >(result));
++- return resultobj;
++-fail:
++- return NULL;
++-}
++-
++-
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsUnknown(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "3"" of type '" "bool""'");
+++ }
+++ arg3 = static_cast< bool >(val3);
+++ ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "4"" of type '" "int""'");
+++ }
+++ arg4 = static_cast< int >(val4);
+++ ecode5 = SWIG_AsVal_float(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "5"" of type '" "float""'");
+++ }
+++ arg5 = static_cast< float >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsIds((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, PyInt_FromLong(static_cast<long>(result[i])));
+++ }
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- int arg2 ;
+++ absl::string_view arg2 ;
+++ bool arg3 ;
+++ int arg4 ;
+++ float arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- bool result;
+++ bool val3 ;
+++ int ecode3 = 0 ;
+++ int val4 ;
+++ int ecode4 = 0 ;
+++ float val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ PyObject *swig_obj[9] ;
+++ std::vector< std::string > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsUnknown", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsPieces", 9, 9, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsUnknown" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsUnknown" "', argument " "2"" of type '" "int""'");
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = absl::string_view(ustring.data(), ustring.size());
+++ }
+++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "3"" of type '" "bool""'");
++ }
++- arg2 = static_cast< int >(val2);
+++ arg3 = static_cast< bool >(val3);
+++ ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "4"" of type '" "int""'");
+++ }
+++ arg4 = static_cast< int >(val4);
+++ ecode5 = SWIG_AsVal_float(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "5"" of type '" "float""'");
+++ }
+++ arg5 = static_cast< float >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
++ {
++ try {
++- result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsUnknown(arg2);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsPieces((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_bool(static_cast< bool >(result));
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ }
+++ }
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsControl(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- int arg2 ;
+++ absl::string_view arg2 ;
+++ bool arg3 ;
+++ int arg4 ;
+++ float arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- bool result;
+++ bool val3 ;
+++ int ecode3 = 0 ;
+++ int val4 ;
+++ int ecode4 = 0 ;
+++ float val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ PyObject *swig_obj[9] ;
+++ sentencepiece::util::bytes result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsControl", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsSerializedProto", 9, 9, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsControl" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsControl" "', argument " "2"" of type '" "int""'");
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = absl::string_view(ustring.data(), ustring.size());
+++ }
+++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "3"" of type '" "bool""'");
++ }
++- arg2 = static_cast< int >(val2);
+++ arg3 = static_cast< bool >(val3);
+++ ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "4"" of type '" "int""'");
+++ }
+++ arg4 = static_cast< int >(val4);
+++ ecode5 = SWIG_AsVal_float(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "5"" of type '" "float""'");
+++ }
+++ arg5 = static_cast< float >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsSerializedProto" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
++ {
++ try {
++- result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsControl(arg2);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_bool(static_cast< bool >(result));
+++ {
+++ resultobj = MakePyOutputBytes(result);
+++ }
++ return resultobj;
++ fail:
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsUnused(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- int arg2 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
+++ int arg3 ;
+++ bool arg4 ;
+++ int arg5 ;
+++ float arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ bool arg10 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- bool result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ bool val4 ;
+++ int ecode4 = 0 ;
+++ int val5 ;
+++ int ecode5 = 0 ;
+++ float val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
+++ std::vector< std::vector< int > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsUnused", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsIdsBatch", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsUnused" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsUnused" "', argument " "2"" of type '" "int""'");
+++ {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "3"" of type '" "int""'");
++ }
++- arg2 = static_cast< int >(val2);
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "4"" of type '" "bool""'");
+++ }
+++ arg4 = static_cast< bool >(val4);
+++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "5"" of type '" "int""'");
+++ }
+++ arg5 = static_cast< int >(val5);
+++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "6"" of type '" "float""'");
+++ }
+++ arg6 = static_cast< float >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsUnused(arg2);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsIdsBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_bool(static_cast< bool >(result));
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyObject *obj = PyList_New(result[i].size());
+++ for (size_t j = 0; j < result[i].size(); ++j) {
+++ PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ }
+++ PyList_SetItem(resultobj, i, obj);
+++ }
+++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_IsByte(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- int arg2 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
+++ int arg3 ;
+++ bool arg4 ;
+++ int arg5 ;
+++ float arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ bool arg10 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- bool result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ bool val4 ;
+++ int ecode4 = 0 ;
+++ int val5 ;
+++ int ecode5 = 0 ;
+++ float val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
+++ std::vector< std::vector< std::string > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_IsByte", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsPiecesBatch", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_IsByte" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "SentencePieceProcessor_IsByte" "', argument " "2"" of type '" "int""'");
+++ {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "3"" of type '" "int""'");
++ }
++- arg2 = static_cast< int >(val2);
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "4"" of type '" "bool""'");
+++ }
+++ arg4 = static_cast< bool >(val4);
+++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "5"" of type '" "int""'");
+++ }
+++ arg5 = static_cast< int >(val5);
+++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "6"" of type '" "float""'");
+++ }
+++ arg6 = static_cast< float >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = (bool)((sentencepiece::SentencePieceProcessor const *)arg1)->IsByte(arg2);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsPiecesBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_bool(static_cast< bool >(result));
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyObject *obj = PyList_New(result[i].size());
+++ for (size_t j = 0; j < result[i].size(); ++j) {
+++ PyList_SetItem(obj, j, MakePyOutputString(result[i][j], input_type));
+++ }
+++ PyList_SetItem(resultobj, i, obj);
+++ }
+++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_unk_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
+++ int arg3 ;
+++ bool arg4 ;
+++ int arg5 ;
+++ float arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ bool arg10 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[1] ;
++- int result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ bool val4 ;
+++ int ecode4 = 0 ;
+++ int val5 ;
+++ int ecode5 = 0 ;
+++ float val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
+++ BytesArray result;
++
++- if (!args) SWIG_fail;
++- swig_obj[0] = args;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsSerializedProtoBatch", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_unk_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "4"" of type '" "bool""'");
+++ }
+++ arg4 = static_cast< bool >(val4);
+++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "5"" of type '" "int""'");
+++ }
+++ arg5 = static_cast< int >(val5);
+++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "6"" of type '" "float""'");
+++ }
+++ arg6 = static_cast< float >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->unk_id();
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_int(static_cast< int >(result));
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, MakePyOutputBytes(result[i]));
+++ }
+++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_bos_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< int > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[1] ;
++- int result;
+++ PyObject *swig_obj[2] ;
+++ std::string result;
++
++- if (!args) SWIG_fail;
++- swig_obj[0] = args;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodeIds", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_bos_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodeIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<int> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<int>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyInt_Check(o)) {
+++ (*out)[i] = static_cast<int>(PyInt_AsLong(o));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
++ {
++ try {
++- result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->bos_id();
+++ result = sentencepiece_SentencePieceProcessor__DecodeIds((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< int > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_int(static_cast< int >(result));
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = MakePyOutputString(result, input_type);
+++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_eos_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< std::string > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[1] ;
++- int result;
+++ PyObject *swig_obj[2] ;
+++ std::string result;
++
++- if (!args) SWIG_fail;
++- swig_obj[0] = args;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodePieces", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_eos_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodePieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<std::string> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<std::string>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i].assign(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
++ }
++- arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++ try {
++- result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->eos_id();
+++ result = sentencepiece_SentencePieceProcessor__DecodePieces((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::string > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_int(static_cast< int >(result));
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = MakePyOutputString(result, input_type);
+++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_pad_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< int > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[1] ;
++- int result;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::util::bytes result;
++
++- if (!args) SWIG_fail;
++- swig_obj[0] = args;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodeIdsAsSerializedProto", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_pad_id" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodeIdsAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<int> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<int>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyInt_Check(o)) {
+++ (*out)[i] = static_cast<int>(PyInt_AsLong(o));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
++ {
++ try {
++- result = (int)((sentencepiece::SentencePieceProcessor const *)arg1)->pad_id();
+++ result = sentencepiece_SentencePieceProcessor__DecodeIdsAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< int > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++ }
++ }
++- resultobj = SWIG_From_int(static_cast< int >(result));
+++ {
+++ resultobj = MakePyOutputBytes(result);
+++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_serialized_model_proto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< std::string > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[1] ;
+++ PyObject *swig_obj[2] ;
++ sentencepiece::util::bytes result;
++
++- if (!args) SWIG_fail;
++- swig_obj[0] = args;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodePiecesAsSerializedProto", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_serialized_model_proto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodePiecesAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<std::string> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<std::string>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i].assign(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->serialized_model_proto();
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::string > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5096,39 +5617,74 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_serialized_model_proto(PyObjec
++ {
++ resultobj = MakePyOutputBytes(result);
++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_LoadFromFile(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
+++ std::vector< std::vector< int > > *arg2 = 0 ;
+++ int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- sentencepiece::util::Status result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ PyObject *swig_obj[3] ;
+++ std::vector< std::string > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_LoadFromFile", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodeIdsBatch", 3, 3, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_LoadFromFile" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodeIdsBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
+++ std::vector<std::vector<int>> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<std::vector<int>>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ PyObject *o2 = PyList_GetItem(o, j);
+++ if (PyInt_Check(o2)) {
+++ (*out)[i][j] = static_cast<int>(PyInt_AsLong(o2));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
++ SWIG_fail;
++ }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = out;
++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__DecodeIdsBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor_LoadFromFile(arg1,arg2);
+++ result = sentencepiece_SentencePieceProcessor__DecodeIdsBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< int > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5136,43 +5692,63 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_LoadFromFile(PyObject *SWIGUNU
++ }
++ }
++ {
++- if (!(&result)->ok()) {
++- SWIG_exception(ToSwigError((&result)->code()), (&result)->ToString().c_str());
+++ PyObject *input_type = resultobj;
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
++ }
++- resultobj = SWIG_From_bool((&result)->ok());
+++ }
+++ {
+++ delete arg2;
++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodeIdsWithCheck(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< int > *arg2 = 0 ;
+++ std::vector< std::vector< int > > *arg2 = 0 ;
+++ int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- std::string result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ PyObject *swig_obj[3] ;
+++ BytesArray result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_DecodeIdsWithCheck", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch", 3, 3, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_DecodeIdsWithCheck" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<int> *out = nullptr;
+++ std::vector<std::vector<int>> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<int>(size);
+++ out = new std::vector<std::vector<int>>(size);
++ for (size_t i = 0; i < size; ++i) {
++ PyObject *o = PyList_GetItem(swig_obj[1], i);
++- if (PyInt_Check(o)) {
++- (*out)[i] = static_cast<int>(PyInt_AsLong(o));
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ PyObject *o2 = PyList_GetItem(o, j);
+++ if (PyInt_Check(o2)) {
+++ (*out)[i][j] = static_cast<int>(PyInt_AsLong(o2));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ }
++ } else {
++- PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ PyErr_SetString(PyExc_TypeError, "not a list");
++ SWIG_fail;
++ }
++ }
++@@ -5182,9 +5758,14 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodeIdsWithCheck(PyObject *S
++ }
++ arg2 = out;
++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor_DecodeIdsWithCheck((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< int > const &)*arg2);
+++ result = sentencepiece_SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< int > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5192,8 +5773,10 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodeIdsWithCheck(PyObject *S
++ }
++ }
++ {
++- PyObject *input_type = resultobj;
++- resultobj = MakePyOutputString(result, input_type);
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, MakePyOutputBytes(result[i]));
+++ }
++ }
++ {
++ delete arg2;
++@@ -5207,32 +5790,46 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< int > *arg2 = 0 ;
+++ std::vector< std::vector< std::string > > *arg2 = 0 ;
+++ int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- PyObject *swig_obj[2] ;
++- sentencepiece::util::bytes result;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ PyObject *swig_obj[3] ;
+++ std::vector< std::string > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodePiecesBatch", 3, 3, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodePiecesBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<int> *out = nullptr;
+++ std::vector<std::vector<std::string>> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<int>(size);
+++ out = new std::vector<std::vector<std::string>>(size);
++ for (size_t i = 0; i < size; ++i) {
++ PyObject *o = PyList_GetItem(swig_obj[1], i);
++- if (PyInt_Check(o)) {
++- (*out)[i] = static_cast<int>(PyInt_AsLong(o));
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ const PyInputString ustring(PyList_GetItem(o, j));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i][j].assign(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
++ } else {
++- PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ PyErr_SetString(PyExc_TypeError,"not a list");
++ SWIG_fail;
++ }
++ }
++@@ -5242,9 +5839,14 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodeIdsAsSerializedProtoWith
++ }
++ arg2 = out;
++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__DecodePiecesBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< int > const &)*arg2);
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< std::string > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5252,7 +5854,11 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_DecodeIdsAsSerializedProtoWith
++ }
++ }
++ {
++- resultobj = MakePyOutputBytes(result);
+++ PyObject *input_type = resultobj;
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ }
++ }
++ {
++ delete arg2;
++@@ -5266,81 +5872,63 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- bool arg3 ;
++- int arg4 ;
++- float arg5 ;
++- bool arg6 ;
++- bool arg7 ;
++- bool arg8 ;
+++ std::vector< std::vector< std::string > > *arg2 = 0 ;
+++ int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- bool val3 ;
+++ int val3 ;
++ int ecode3 = 0 ;
++- int val4 ;
++- int ecode4 = 0 ;
++- float val5 ;
++- int ecode5 = 0 ;
++- bool val6 ;
++- int ecode6 = 0 ;
++- bool val7 ;
++- int ecode7 = 0 ;
++- bool val8 ;
++- int ecode8 = 0 ;
++- PyObject *swig_obj[8] ;
++- std::vector< int > result;
+++ PyObject *swig_obj[3] ;
+++ BytesArray result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsIds", 8, 8, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch", 3, 3, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
++- }
++- arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++- ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "3"" of type '" "bool""'");
++- }
++- arg3 = static_cast< bool >(val3);
++- ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "4"" of type '" "int""'");
++- }
++- arg4 = static_cast< int >(val4);
++- ecode5 = SWIG_AsVal_float(swig_obj[4], &val5);
++- if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "5"" of type '" "float""'");
++- }
++- arg5 = static_cast< float >(val5);
++- ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++- if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "6"" of type '" "bool""'");
++- }
++- arg6 = static_cast< bool >(val6);
++- ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
++- if (!SWIG_IsOK(ecode7)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "7"" of type '" "bool""'");
++- }
++- arg7 = static_cast< bool >(val7);
++- ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
++- if (!SWIG_IsOK(ecode8)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsIds" "', argument " "8"" of type '" "bool""'");
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<std::vector<std::string>> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<std::vector<std::string>>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ const PyInputString ustring(PyList_GetItem(o, j));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i][j].assign(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch" "', argument " "3"" of type '" "int""'");
++ }
++- arg8 = static_cast< bool >(val8);
+++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__EncodeAsIds(arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8);
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< std::string > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5350,49 +5938,49 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIds(PyObject *SWIGUNU
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, PyInt_FromLong(static_cast<long>(result[i])));
+++ PyList_SetItem(resultobj, i, MakePyOutputBytes(result[i]));
++ }
++ }
+++ {
+++ delete arg2;
+++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
++- bool arg3 ;
++- int arg4 ;
++- float arg5 ;
+++ int arg3 ;
+++ bool arg4 ;
+++ bool arg5 ;
++ bool arg6 ;
++ bool arg7 ;
++- bool arg8 ;
++- bool arg9 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- bool val3 ;
+++ int val3 ;
++ int ecode3 = 0 ;
++- int val4 ;
+++ bool val4 ;
++ int ecode4 = 0 ;
++- float val5 ;
+++ bool val5 ;
++ int ecode5 = 0 ;
++ bool val6 ;
++ int ecode6 = 0 ;
++ bool val7 ;
++ int ecode7 = 0 ;
++- bool val8 ;
++- int ecode8 = 0 ;
++- bool val9 ;
++- int ecode9 = 0 ;
++- PyObject *swig_obj[9] ;
++- std::vector< std::string > result;
+++ PyObject *swig_obj[7] ;
+++ std::vector< std::vector< int > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsPieces", 9, 9, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__NBestEncodeAsIds", 7, 7, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5404,44 +5992,34 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPieces(PyObject *SWIG
++ resultobj = ustring.input_type();
++ arg2 = absl::string_view(ustring.data(), ustring.size());
++ }
++- ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "3"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "3"" of type '" "int""'");
++ }
++- arg3 = static_cast< bool >(val3);
++- ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
++ if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "4"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "4"" of type '" "bool""'");
++ }
++- arg4 = static_cast< int >(val4);
++- ecode5 = SWIG_AsVal_float(swig_obj[4], &val5);
+++ arg4 = static_cast< bool >(val4);
+++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++ if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "5"" of type '" "float""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "5"" of type '" "bool""'");
++ }
++- arg5 = static_cast< float >(val5);
+++ arg5 = static_cast< bool >(val5);
++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++ if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "6"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "6"" of type '" "bool""'");
++ }
++ arg6 = static_cast< bool >(val6);
++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
++ if (!SWIG_IsOK(ecode7)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "7"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "7"" of type '" "bool""'");
++ }
++ arg7 = static_cast< bool >(val7);
++- ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
++- if (!SWIG_IsOK(ecode8)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "8"" of type '" "bool""'");
++- }
++- arg8 = static_cast< bool >(val8);
++- ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
++- if (!SWIG_IsOK(ecode9)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsPieces" "', argument " "9"" of type '" "bool""'");
++- }
++- arg9 = static_cast< bool >(val9);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__EncodeAsPieces(arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9);
+++ result = sentencepiece_SentencePieceProcessor__NBestEncodeAsIds((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5449,10 +6027,13 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPieces(PyObject *SWIG
++ }
++ }
++ {
++- PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ PyObject *obj = PyList_New(result[i].size());
+++ for (size_t j = 0; j < result[i].size(); ++j) {
+++ PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ }
+++ PyList_SetItem(resultobj, i, obj);
++ }
++ }
++ return resultobj;
++@@ -5461,7 +6042,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
++@@ -5469,6 +6050,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SW
++ bool arg4 ;
++ bool arg5 ;
++ bool arg6 ;
+++ bool arg7 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++ int val3 ;
++@@ -5479,13 +6061,15 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SW
++ int ecode5 = 0 ;
++ bool val6 ;
++ int ecode6 = 0 ;
++- PyObject *swig_obj[6] ;
++- std::vector< std::vector< int > > result;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ PyObject *swig_obj[7] ;
+++ std::vector< std::vector< std::string > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__NBestEncodeAsIds", 6, 6, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__NBestEncodeAsPieces", 7, 7, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5499,27 +6083,32 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SW
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "3"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "3"" of type '" "int""'");
++ }
++ arg3 = static_cast< int >(val3);
++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
++ if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "4"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "4"" of type '" "bool""'");
++ }
++ arg4 = static_cast< bool >(val4);
++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++ if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "5"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "5"" of type '" "bool""'");
++ }
++ arg5 = static_cast< bool >(val5);
++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++ if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__NBestEncodeAsIds" "', argument " "6"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "6"" of type '" "bool""'");
++ }
++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__NBestEncodeAsIds(arg1,arg2,arg3,arg4,arg5,arg6);
+++ result = sentencepiece_SentencePieceProcessor__NBestEncodeAsPieces((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5527,11 +6116,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SW
++ }
++ }
++ {
+++ PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].size());
++ for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ PyList_SetItem(obj, j, MakePyOutputString(result[i][j], input_type));
++ }
++ PyList_SetItem(resultobj, i, obj);
++ }
++@@ -5542,7 +6132,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
++@@ -5564,12 +6154,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject
++ bool val7 ;
++ int ecode7 = 0 ;
++ PyObject *swig_obj[7] ;
++- std::vector< std::vector< std::string > > result;
+++ sentencepiece::util::bytes result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__NBestEncodeAsPieces", 7, 7, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__NBestEncodeAsSerializedProto", 7, 7, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__NBestEncodeAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5583,32 +6173,32 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "3"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__NBestEncodeAsSerializedProto" "', argument " "3"" of type '" "int""'");
++ }
++ arg3 = static_cast< int >(val3);
++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
++ if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "4"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__NBestEncodeAsSerializedProto" "', argument " "4"" of type '" "bool""'");
++ }
++ arg4 = static_cast< bool >(val4);
++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++ if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "5"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__NBestEncodeAsSerializedProto" "', argument " "5"" of type '" "bool""'");
++ }
++ arg5 = static_cast< bool >(val5);
++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++ if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "6"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__NBestEncodeAsSerializedProto" "', argument " "6"" of type '" "bool""'");
++ }
++ arg6 = static_cast< bool >(val6);
++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
++ if (!SWIG_IsOK(ecode7)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__NBestEncodeAsPieces" "', argument " "7"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__NBestEncodeAsSerializedProto" "', argument " "7"" of type '" "bool""'");
++ }
++ arg7 = static_cast< bool >(val7);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__NBestEncodeAsPieces(arg1,arg2,arg3,arg4,arg5,arg6,arg7);
+++ result = sentencepiece_SentencePieceProcessor__NBestEncodeAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5616,15 +6206,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject
++ }
++ }
++ {
++- PyObject *input_type = resultobj;
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].size());
++- for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i][j], input_type));
++- }
++- PyList_SetItem(resultobj, i, obj);
++- }
+++ resultobj = MakePyOutputBytes(result);
++ }
++ return resultobj;
++ fail:
++@@ -5643,6 +6225,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds(PyO
++ bool arg7 ;
++ bool arg8 ;
++ bool arg9 ;
+++ bool arg10 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++ int val3 ;
++@@ -5659,13 +6242,15 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds(PyO
++ int ecode8 = 0 ;
++ bool val9 ;
++ int ecode9 = 0 ;
++- PyObject *swig_obj[9] ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
++ std::vector< std::pair< std::vector< int >,float > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__SampleEncodeAndScoreAsIds", 9, 9, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__SampleEncodeAndScoreAsIds", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5712,9 +6297,14 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds(PyO
++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsIds" "', argument " "9"" of type '" "bool""'");
++ }
++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsIds" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds(arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9);
+++ result = sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5773,7 +6363,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(
++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__SampleEncodeAndScoreAsPieces", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5827,7 +6417,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(
++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ result = sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsPieces((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5851,6 +6441,133 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__CalculateEntropy(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ absl::string_view arg2 ;
+++ float arg3 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ float val3 ;
+++ int ecode3 = 0 ;
+++ PyObject *swig_obj[3] ;
+++ float result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__CalculateEntropy", 3, 3, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = absl::string_view(ustring.data(), ustring.size());
+++ }
+++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__CalculateEntropy" "', argument " "3"" of type '" "float""'");
+++ }
+++ arg3 = static_cast< float >(val3);
+++ {
+++ try {
+++ result = (float)sentencepiece_SentencePieceProcessor__CalculateEntropy(arg1,arg2,arg3);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_float(static_cast< float >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__CalculateEntropyBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
+++ float arg3 ;
+++ int arg4 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ float val3 ;
+++ int ecode3 = 0 ;
+++ int val4 ;
+++ int ecode4 = 0 ;
+++ PyObject *swig_obj[4] ;
+++ std::vector< float > result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__CalculateEntropyBatch", 4, 4, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__CalculateEntropyBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__CalculateEntropyBatch" "', argument " "3"" of type '" "float""'");
+++ }
+++ arg3 = static_cast< float >(val3);
+++ ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__CalculateEntropyBatch" "', argument " "4"" of type '" "int""'");
+++ }
+++ arg4 = static_cast< int >(val4);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__CalculateEntropyBatch(arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SetItem(resultobj, i, PyFloat_FromDouble(static_cast<double>(result[i])));
+++ }
+++ }
+++ {
+++ delete arg2;
+++ }
+++ return resultobj;
+++fail:
+++ {
+++ delete arg2;
+++ }
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *SentencePieceProcessor_swigregister(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *obj;
++ if (!SWIG_Python_UnpackTuple(args, "swigregister", 1, 1, &obj)) return NULL;
++@@ -6191,20 +6908,9 @@ static PyMethodDef SwigMethods[] = {
++ { "SentencePieceProcessor_SetVocabulary", _wrap_SentencePieceProcessor_SetVocabulary, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_ResetVocabulary", _wrap_SentencePieceProcessor_ResetVocabulary, METH_O, NULL},
++ { "SentencePieceProcessor_LoadVocabulary", _wrap_SentencePieceProcessor_LoadVocabulary, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_EncodeAsPieces", _wrap_SentencePieceProcessor_EncodeAsPieces, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_EncodeAsIds", _wrap_SentencePieceProcessor_EncodeAsIds, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_NBestEncodeAsPieces", _wrap_SentencePieceProcessor_NBestEncodeAsPieces, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_NBestEncodeAsIds", _wrap_SentencePieceProcessor_NBestEncodeAsIds, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_SampleEncodeAsPieces", _wrap_SentencePieceProcessor_SampleEncodeAsPieces, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_SampleEncodeAsIds", _wrap_SentencePieceProcessor_SampleEncodeAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_SampleEncodeAndScoreAsPieces", _wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_SampleEncodeAndScoreAsIds", _wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_DecodePieces", _wrap_SentencePieceProcessor_DecodePieces, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_CalculateEntropy", _wrap_SentencePieceProcessor_CalculateEntropy, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_EncodeAsSerializedProto", _wrap_SentencePieceProcessor_EncodeAsSerializedProto, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_SampleEncodeAsSerializedProto", _wrap_SentencePieceProcessor_SampleEncodeAsSerializedProto, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_NBestEncodeAsSerializedProto", _wrap_SentencePieceProcessor_NBestEncodeAsSerializedProto, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_DecodePiecesAsSerializedProto", _wrap_SentencePieceProcessor_DecodePiecesAsSerializedProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_GetPieceSize", _wrap_SentencePieceProcessor_GetPieceSize, METH_O, NULL},
++ { "SentencePieceProcessor_PieceToId", _wrap_SentencePieceProcessor_PieceToId, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_IdToPiece", _wrap_SentencePieceProcessor_IdToPiece, METH_VARARGS, NULL},
++@@ -6219,14 +6925,27 @@ static PyMethodDef SwigMethods[] = {
++ { "SentencePieceProcessor_pad_id", _wrap_SentencePieceProcessor_pad_id, METH_O, NULL},
++ { "SentencePieceProcessor_serialized_model_proto", _wrap_SentencePieceProcessor_serialized_model_proto, METH_O, NULL},
++ { "SentencePieceProcessor_LoadFromFile", _wrap_SentencePieceProcessor_LoadFromFile, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_DecodeIdsWithCheck", _wrap_SentencePieceProcessor_DecodeIdsWithCheck, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck", _wrap_SentencePieceProcessor_DecodeIdsAsSerializedProtoWithCheck, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsIds", _wrap_SentencePieceProcessor__EncodeAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsPieces", _wrap_SentencePieceProcessor__EncodeAsPieces, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__EncodeAsSerializedProto", _wrap_SentencePieceProcessor__EncodeAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__EncodeAsIdsBatch", _wrap_SentencePieceProcessor__EncodeAsIdsBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__EncodeAsPiecesBatch", _wrap_SentencePieceProcessor__EncodeAsPiecesBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__EncodeAsSerializedProtoBatch", _wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodeIds", _wrap_SentencePieceProcessor__DecodeIds, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodePieces", _wrap_SentencePieceProcessor__DecodePieces, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodeIdsAsSerializedProto", _wrap_SentencePieceProcessor__DecodeIdsAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodePiecesAsSerializedProto", _wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodeIdsBatch", _wrap_SentencePieceProcessor__DecodeIdsBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch", _wrap_SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodePiecesBatch", _wrap_SentencePieceProcessor__DecodePiecesBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch", _wrap_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__NBestEncodeAsIds", _wrap_SentencePieceProcessor__NBestEncodeAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__NBestEncodeAsPieces", _wrap_SentencePieceProcessor__NBestEncodeAsPieces, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__NBestEncodeAsSerializedProto", _wrap_SentencePieceProcessor__NBestEncodeAsSerializedProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__SampleEncodeAndScoreAsIds", _wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__SampleEncodeAndScoreAsPieces", _wrap_SentencePieceProcessor__SampleEncodeAndScoreAsPieces, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__CalculateEntropy", _wrap_SentencePieceProcessor__CalculateEntropy, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__CalculateEntropyBatch", _wrap_SentencePieceProcessor__CalculateEntropyBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_swigregister", SentencePieceProcessor_swigregister, METH_O, NULL},
++ { "SentencePieceProcessor_swiginit", SentencePieceProcessor_swiginit, METH_VARARGS, NULL},
++ { "SetRandomGeneratorSeed", _wrap_SetRandomGeneratorSeed, METH_O, NULL},
++@@ -6252,8 +6971,11 @@ static swig_type_info _swigt__p_sentencepiece__SentencePieceProcessor = {"_p_sen
++ static swig_type_info _swigt__p_sentencepiece__SentencePieceTrainer = {"_p_sentencepiece__SentencePieceTrainer", "sentencepiece::SentencePieceTrainer *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__string = {"_p_std__string", "sentencepiece::util::bytes *|std::string *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__unordered_mapT_std__string_std__string_t = {"_p_std__unordered_mapT_std__string_std__string_t", "std::unordered_map< std::string,std::string > *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_std__vectorT_absl__string_view_t = {"_p_std__vectorT_absl__string_view_t", "std::vector< absl::string_view > *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__vectorT_int_t = {"_p_std__vectorT_int_t", "std::vector< int > *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__vectorT_std__string_t = {"_p_std__vectorT_std__string_t", "std::vector< std::string > *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_std__vectorT_std__vectorT_int_t_t = {"_p_std__vectorT_std__vectorT_int_t_t", "std::vector< std::vector< int > > *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_std__vectorT_std__vectorT_std__string_t_t = {"_p_std__vectorT_std__vectorT_std__string_t_t", "std::vector< std::vector< std::string > > *", 0, 0, (void*)0, 0};
++
++ static swig_type_info *swig_type_initial[] = {
++ &_swigt__p_char,
++@@ -6262,8 +6984,11 @@ static swig_type_info *swig_type_initial[] = {
++ &_swigt__p_sentencepiece__SentencePieceTrainer,
++ &_swigt__p_std__string,
++ &_swigt__p_std__unordered_mapT_std__string_std__string_t,
+++ &_swigt__p_std__vectorT_absl__string_view_t,
++ &_swigt__p_std__vectorT_int_t,
++ &_swigt__p_std__vectorT_std__string_t,
+++ &_swigt__p_std__vectorT_std__vectorT_int_t_t,
+++ &_swigt__p_std__vectorT_std__vectorT_std__string_t_t,
++ };
++
++ static swig_cast_info _swigc__p_char[] = { {&_swigt__p_char, 0, 0, 0},{0, 0, 0, 0}};
++@@ -6272,8 +6997,11 @@ static swig_cast_info _swigc__p_sentencepiece__SentencePieceProcessor[] = { {&_
++ static swig_cast_info _swigc__p_sentencepiece__SentencePieceTrainer[] = { {&_swigt__p_sentencepiece__SentencePieceTrainer, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__string[] = { {&_swigt__p_std__string, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__unordered_mapT_std__string_std__string_t[] = { {&_swigt__p_std__unordered_mapT_std__string_std__string_t, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_std__vectorT_absl__string_view_t[] = { {&_swigt__p_std__vectorT_absl__string_view_t, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__vectorT_int_t[] = { {&_swigt__p_std__vectorT_int_t, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__vectorT_std__string_t[] = { {&_swigt__p_std__vectorT_std__string_t, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_std__vectorT_std__vectorT_int_t_t[] = { {&_swigt__p_std__vectorT_std__vectorT_int_t_t, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_std__vectorT_std__vectorT_std__string_t_t[] = { {&_swigt__p_std__vectorT_std__vectorT_std__string_t_t, 0, 0, 0},{0, 0, 0, 0}};
++
++ static swig_cast_info *swig_cast_initial[] = {
++ _swigc__p_char,
++@@ -6282,8 +7010,11 @@ static swig_cast_info *swig_cast_initial[] = {
++ _swigc__p_sentencepiece__SentencePieceTrainer,
++ _swigc__p_std__string,
++ _swigc__p_std__unordered_mapT_std__string_std__string_t,
+++ _swigc__p_std__vectorT_absl__string_view_t,
++ _swigc__p_std__vectorT_int_t,
++ _swigc__p_std__vectorT_std__string_t,
+++ _swigc__p_std__vectorT_std__vectorT_int_t_t,
+++ _swigc__p_std__vectorT_std__vectorT_std__string_t_t,
++ };
++
++
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index b747e81..99e36f3 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -15,7 +15,6 @@
++ # See the License for the specific language governing permissions and
++ # limitations under the License.!
++
++-import codecs
++ import io
++ import sentencepiece as spm
++ import unittest
++@@ -62,6 +61,17 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ piece = self.sp_.IdToPiece(i)
++ self.assertEqual(i, self.sp_.PieceToId(piece))
++
+++ self.assertEqual(1000, self.sp_.get_piece_size())
+++ self.assertEqual(0, self.sp_.piece_to_id('<unk>'))
+++ self.assertEqual(1, self.sp_.piece_to_id('<s>'))
+++ self.assertEqual(2, self.sp_.piece_to_id('</s>'))
+++ self.assertEqual('<unk>', self.sp_.id_to_piece(0))
+++ self.assertEqual('<s>', self.sp_.id_to_piece(1))
+++ self.assertEqual('</s>', self.sp_.id_to_piece(2))
+++ for i in range(self.sp_.get_piece_size()):
+++ piece = self.sp_.id_to_piece(i)
+++ self.assertEqual(i, self.sp_.piece_to_id(piece))
+++
++ def test_roundtrip(self):
++ text = 'I saw a girl with a telescope.'
++ ids = self.sp_.EncodeAsIds(text)
++@@ -82,6 +92,34 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(
++ text, self.sp_.DecodeIds(self.sp_.SampleEncodeAsIds(text, -1, 0.5)))
++
+++ ids2 = self.sp_.encode_as_ids(text)
+++ pieces3 = self.sp_.encode_as_pieces(text)
+++ pieces4 = self.sp_.nbest_encode_as_pieces(text, 10)[0]
+++ self.assertEqual(pieces3, pieces4)
+++ self.assertEqual(pieces1, pieces3)
+++ self.assertEqual(ids, ids2)
+++ self.assertEqual(text, self.sp_.decode_pieces(pieces3))
+++ self.assertEqual(text, self.sp_.decode_ids(ids2))
+++ for n in range(100):
+++ self.assertEqual(
+++ text,
+++ self.sp_.decode_pieces(
+++ self.sp_.sample_encode_as_pieces(text, 64, 0.5)))
+++ self.assertEqual(
+++ text,
+++ self.sp_.decode_pieces(
+++ self.sp_.sample_encode_as_pieces(text, -1, 0.5)))
+++ self.assertEqual(
+++ text,
+++ self.sp_.decode_ids(self.sp_.sample_encode_as_ids(text, 64, 0.5)))
+++ self.assertEqual(
+++ text,
+++ self.sp_.decode_ids(self.sp_.sample_encode_as_ids(text, -1, 0.5)))
+++
+++ self.assertEqual(
+++ self.sp_.calculate_entropy(text, 0.1),
+++ self.sp_.CalculateEntropy(text, 0.1))
+++
++ def test_ja_load(self):
++ self.assertEqual(8000, self.jasp_.GetPieceSize())
++ self.assertEqual(0, self.jasp_.PieceToId('<unk>'))
++@@ -94,6 +132,17 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ piece = self.jasp_.IdToPiece(i)
++ self.assertEqual(i, self.jasp_.PieceToId(piece))
++
+++ self.assertEqual(8000, self.jasp_.get_piece_size())
+++ self.assertEqual(0, self.jasp_.piece_to_id('<unk>'))
+++ self.assertEqual(1, self.jasp_.piece_to_id('<s>'))
+++ self.assertEqual(2, self.jasp_.piece_to_id('</s>'))
+++ self.assertEqual('<unk>', self.jasp_.id_to_piece(0))
+++ self.assertEqual('<s>', self.jasp_.id_to_piece(1))
+++ self.assertEqual('</s>', self.jasp_.id_to_piece(2))
+++ for i in range(self.jasp_.get_piece_size()):
+++ piece = self.jasp_.id_to_piece(i)
+++ self.assertEqual(i, self.jasp_.piece_to_id(piece))
+++
++ def test_ja_roundtrip(self):
++ text = '清水寺は京都にある。'
++ ids = self.jasp_.EncodeAsIds(text)
++@@ -112,40 +161,27 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.jasp_.DecodePieces(
++ self.jasp_.SampleEncodeAsPieces(text, -1, 0.5)))
++
++- def test_unicode_roundtrip(self):
++- text = u'I saw a girl with a telescope.'
++- ids = self.sp_.EncodeAsIds(text)
++- pieces = self.sp_.EncodeAsPieces(text)
++- self.assertEqual(text, self.sp_.DecodePieces(pieces))
++- self.assertEqual(text, self.sp_.DecodeIds(ids))
++- # python2 returns `str`.
++- if sys.version_info < (3, 0, 0):
++- text = text.encode('utf-8')
++- self.assertEqual(text, self.sp_.DecodeIds(ids))
++- self.assertEqual(text, self.sp_.DecodePieces(pieces))
++-
++- def test_unicode_ja_roundtrip(self):
++- text = u'清水寺は京都にある。'
++- ids = self.jasp_.EncodeAsIds(text)
++- pieces = self.jasp_.EncodeAsPieces(text)
++- self.assertEqual(text, self.jasp_.DecodePieces(pieces))
++- # python2 returns `str`.
++- if sys.version_info < (3, 0, 0):
++- text = text.encode('utf-8')
++- self.assertEqual(text, self.jasp_.DecodeIds(ids))
++-
++- def test_pickle(self):
++- with open('sp.pickle', 'wb') as f:
++- pickle.dump(self.sp_, f)
++-
++- id1 = self.sp_.encode('hello world.', out_type=int)
++-
++- with open('sp.pickle', 'rb') as f:
++- sp = pickle.load(f)
++-
++- id2 = sp.encode('hello world.', out_type=int)
+++ ids2 = self.jasp_.encode_as_ids(text)
+++ pieces3 = self.jasp_.encode_as_pieces(text)
+++ pieces4 = self.jasp_.nbest_encode_as_pieces(text, 10)[0]
+++ self.assertEqual(pieces3, pieces4)
+++ self.assertEqual(pieces1, pieces3)
+++ self.assertEqual(ids, ids2)
+++ self.assertEqual(text, self.jasp_.decode_pieces(pieces1))
+++ self.assertEqual(text, self.jasp_.decode_ids(ids2))
+++ for n in range(100):
+++ self.assertEqual(
+++ text,
+++ self.jasp_.decode_pieces(
+++ self.jasp_.sample_encode_as_pieces(text, 64, 0.5)))
+++ self.assertEqual(
+++ text,
+++ self.jasp_.decode_pieces(
+++ self.jasp_.sample_encode_as_pieces(text, -1, 0.5)))
++
++- self.assertEqual(id1, id2)
+++ self.assertEqual(
+++ self.jasp_.calculate_entropy(text, 0.1),
+++ self.jasp_.CalculateEntropy(text, 0.1))
++
++ def test_train(self):
++ spm.SentencePieceTrainer.Train('--input=' +
++@@ -153,37 +189,45 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ ' --model_prefix=m --vocab_size=1000')
++ sp = spm.SentencePieceProcessor()
++ sp.Load('m.model')
++- with codecs.open(
++- os.path.join(data_dir, 'botchan.txt'), 'r', encoding='utf-8') as file:
+++ with open(os.path.join(data_dir, 'botchan.txt'), 'r') as file:
++ for line in file:
++ sp.DecodePieces(sp.EncodeAsPieces(line))
++ sp.DecodeIds(sp.EncodeAsIds(line))
++
++- def test_train(self):
+++ def test_train_iterator(self):
++ spm.SentencePieceTrainer.Train('--input=' +
++ os.path.join(data_dir, 'botchan.txt') +
++ ' --model_prefix=m --vocab_size=1000')
++ # Load as 'rb' for Python3.5/2.7.
++- is1 = open(os.path.join(data_dir, 'botchan.txt'), 'rb')
++- is2 = open(os.path.join(data_dir, 'botchan.txt'), 'rb')
++ os1 = io.BytesIO()
++ os2 = io.BytesIO()
++
+++ # suppress logging (redirect to /dev/null)
++ spm.SentencePieceTrainer.train(
++ input=os.path.join(data_dir, 'botchan.txt'),
++ model_prefix='m',
++- vocab_size=1000)
+++ vocab_size=1000,
+++ logstream=open(os.devnull, 'w'))
++
++- spm.SentencePieceTrainer.train(
++- sentence_iterator=is1, model_prefix='m', vocab_size=1000)
+++ with open(os.path.join(data_dir, 'botchan.txt'), 'rb') as is1:
+++ spm.SentencePieceTrainer.train(
+++ sentence_iterator=is1,
+++ model_prefix='m',
+++ vocab_size=1000,
+++ logstream=open(os.devnull, 'w'))
++
++ spm.SentencePieceTrainer.train(
++ input=os.path.join(data_dir, 'botchan.txt'),
++ model_writer=os1,
++- vocab_size=1000)
+++ vocab_size=1000,
+++ logstream=open(os.devnull, 'w'))
++
++- spm.SentencePieceTrainer.train(
++- sentence_iterator=is2, model_writer=os2, vocab_size=1000)
+++ with open(os.path.join(data_dir, 'botchan.txt'), 'rb') as is2:
+++ spm.SentencePieceTrainer.train(
+++ sentence_iterator=is2,
+++ model_writer=os2,
+++ vocab_size=1000,
+++ logstream=open(os.devnull, 'w'))
++
++ sp1 = spm.SentencePieceProcessor(model_proto=os1.getvalue())
++ sp2 = spm.SentencePieceProcessor(model_proto=os2.getvalue())
++@@ -200,127 +244,37 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ logstream=open(os.devnull, 'w'))
++ sp = spm.SentencePieceProcessor()
++ sp.Load('m.model')
++- with codecs.open(
+++ with open(
++ os.path.join(data_dir, 'botchan.txt'), 'r', encoding='utf-8') as file:
++ for line in file:
++ sp.DecodePieces(sp.EncodeAsPieces(line))
++ sp.DecodeIds(sp.EncodeAsIds(line))
++
++- # snake case API.
++- def test_load_snake(self):
++- self.assertEqual(1000, self.sp_.get_piece_size())
++- self.assertEqual(0, self.sp_.piece_to_id('<unk>'))
++- self.assertEqual(1, self.sp_.piece_to_id('<s>'))
++- self.assertEqual(2, self.sp_.piece_to_id('</s>'))
++- self.assertEqual('<unk>', self.sp_.id_to_piece(0))
++- self.assertEqual('<s>', self.sp_.id_to_piece(1))
++- self.assertEqual('</s>', self.sp_.id_to_piece(2))
++- for i in range(self.sp_.get_piece_size()):
++- piece = self.sp_.id_to_piece(i)
++- self.assertEqual(i, self.sp_.piece_to_id(piece))
++-
++- def test_roundtrip_snake(self):
++- text = 'I saw a girl with a telescope.'
++- ids = self.sp_.encode_as_ids(text)
++- pieces1 = self.sp_.encode_as_pieces(text)
++- pieces2 = self.sp_.nbest_encode_as_pieces(text, 10)[0]
++- self.assertEqual(pieces1, pieces2)
++- self.assertEqual(text, self.sp_.decode_pieces(pieces1))
++- self.assertEqual(text, self.sp_.decode_ids(ids))
++- for n in range(100):
++- self.assertEqual(
++- text,
++- self.sp_.decode_pieces(
++- self.sp_.sample_encode_as_pieces(text, 64, 0.5)))
++- self.assertEqual(
++- text,
++- self.sp_.decode_pieces(
++- self.sp_.sample_encode_as_pieces(text, -1, 0.5)))
++- self.assertEqual(
++- text,
++- self.sp_.decode_ids(self.sp_.sample_encode_as_ids(text, 64, 0.5)))
++- self.assertEqual(
++- text,
++- self.sp_.decode_ids(self.sp_.sample_encode_as_ids(text, -1, 0.5)))
++-
++- def test_ja_load_snake(self):
++- self.assertEqual(8000, self.jasp_.get_piece_size())
++- self.assertEqual(0, self.jasp_.piece_to_id('<unk>'))
++- self.assertEqual(1, self.jasp_.piece_to_id('<s>'))
++- self.assertEqual(2, self.jasp_.piece_to_id('</s>'))
++- self.assertEqual('<unk>', self.jasp_.id_to_piece(0))
++- self.assertEqual('<s>', self.jasp_.id_to_piece(1))
++- self.assertEqual('</s>', self.jasp_.id_to_piece(2))
++- for i in range(self.jasp_.get_piece_size()):
++- piece = self.jasp_.id_to_piece(i)
++- self.assertEqual(i, self.jasp_.piece_to_id(piece))
++-
++- def test_ja_roundtrip_snake(self):
++- text = '清水寺は京都にある。'
++- ids = self.jasp_.encode_as_ids(text)
++- pieces1 = self.jasp_.encode_as_pieces(text)
++- pieces2 = self.jasp_.nbest_encode_as_pieces(text, 10)[0]
++- self.assertEqual(pieces1, pieces2)
++- self.assertEqual(text, self.jasp_.decode_pieces(pieces1))
++- self.assertEqual(text, self.jasp_.decode_ids(ids))
++- for n in range(100):
++- self.assertEqual(
++- text,
++- self.jasp_.decode_pieces(
++- self.jasp_.sample_encode_as_pieces(text, 64, 0.5)))
++- self.assertEqual(
++- text,
++- self.jasp_.decode_pieces(
++- self.jasp_.sample_encode_as_pieces(text, -1, 0.5)))
++-
++- def test_unicode_roundtrip_snake(self):
++- text = u'I saw a girl with a telescope.'
++- ids = self.sp_.encode_as_ids(text)
++- pieces = self.sp_.encode_as_pieces(text)
++- self.assertEqual(text, self.sp_.decode_pieces(pieces))
++- # python2 returns `str`.
++- if sys.version_info < (3, 0, 0):
++- text = text.encode('utf-8')
++- self.assertEqual(text, self.sp_.decode_ids(ids))
++-
++- def test_unicode_ja_roundtrip_snake(self):
++- text = u'清水寺は京都にある。'
++- ids = self.jasp_.encode_as_ids(text)
++- pieces = self.jasp_.encode_as_pieces(text)
++- self.assertEqual(text, self.jasp_.decode_pieces(pieces))
++- # python2 returns `str`.
++- if sys.version_info < (3, 0, 0):
++- text = text.encode('utf-8')
++- self.assertEqual(text, self.jasp_.decode_ids(ids))
++-
++- def test_train_snake(self):
++- spm.SentencePieceTrainer.train('--input=' +
++- os.path.join(data_dir, 'botchan.txt') +
++- ' --model_prefix=m --vocab_size=1000')
++- sp = spm.SentencePieceProcessor()
++- sp.load('m.model')
++- with codecs.open(
++- os.path.join(data_dir, 'botchan.txt'), 'r', encoding='utf-8') as file:
++- for line in file:
++- sp.decode_pieces(sp.encode_as_pieces(line))
++- sp.decode_ids(sp.encode_as_ids(line))
++-
++ def test_serialized_proto(self):
++- text = u'I saw a girl with a telescope.'
++- self.assertNotEqual('', self.sp_.EncodeAsSerializedProto(text))
++- self.assertNotEqual('',
++- self.sp_.SampleEncodeAsSerializedProto(text, 10, 0.2))
++- self.assertNotEqual('', self.sp_.NBestEncodeAsSerializedProto(text, 10))
++- self.assertNotEqual('',
++- self.sp_.DecodePiecesAsSerializedProto(['foo', 'bar']))
++- self.assertNotEqual('', self.sp_.DecodeIdsAsSerializedProto([20, 30]))
++- self.assertNotEqual('', self.sp_.encode_as_serialized_proto(text))
++- self.assertNotEqual(
++- '', self.sp_.sample_encode_as_serialized_proto(text, 10, 0.2))
++- self.assertNotEqual('', self.sp_.nbest_encode_as_serialized_proto(text, 10))
++- self.assertNotEqual(
++- '', self.sp_.decode_pieces_as_serialized_proto(['foo', 'bar']))
++- self.assertNotEqual('', self.sp_.decode_ids_as_serialized_proto([20, 30]))
+++ text = 'I saw a girl with a telescope.'
+++ s1 = self.sp_.EncodeAsSerializedProto(text)
+++ s2 = self.sp_.SampleEncodeAsSerializedProto(text, 10, 0.2)
+++ s3 = self.sp_.NBestEncodeAsSerializedProto(text, 10)
+++ s4 = self.sp_.DecodePiecesAsSerializedProto(['foo', 'bar'])
+++ s5 = self.sp_.DecodeIdsAsSerializedProto([20, 30])
+++
+++ t1 = self.sp_.encode_as_serialized_proto(text)
+++ t2 = self.sp_.sample_encode_as_serialized_proto(text, 10, 0.2)
+++ t3 = self.sp_.nbest_encode_as_serialized_proto(text, 10)
+++ t4 = self.sp_.decode_pieces_as_serialized_proto(['foo', 'bar'])
+++ t5 = self.sp_.decode_ids_as_serialized_proto([20, 30])
+++
+++ self.assertEqual(type(s1), bytes)
+++ self.assertEqual(type(s2), bytes)
+++ self.assertEqual(type(t2), bytes)
+++ self.assertEqual(type(s3), bytes)
+++ self.assertEqual(type(s4), bytes)
+++ self.assertEqual(type(s5), bytes)
+++
+++ self.assertEqual(s1, t1)
+++ self.assertEqual(s3, t3)
+++ self.assertEqual(s4, t4)
+++ self.assertEqual(s5, t5)
++
++ def test_new_api(self):
++ sp = spm.SentencePieceProcessor(
++@@ -331,19 +285,33 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ ids2 = self.sp_.EncodeAsIds(text2)
++ pieces = self.sp_.EncodeAsPieces(text)
++ pieces2 = self.sp_.EncodeAsPieces(text2)
++- self.assertEqual(sp.encode(text), ids)
+++ protos = self.sp_.EncodeAsSerializedProto(text)
+++ proto2 = self.sp_.EncodeAsSerializedProto(text2)
+++
+++ self.assertEqual(sp.encode(text, out_type=int), ids)
++ self.assertEqual(sp.encode(text, out_type=str), pieces)
+++ self.assertEqual(sp.encode(text, out_type='proto'), protos)
+++
+++ self.assertEqual(sp.encode([text], out_type=int), [ids])
+++ self.assertEqual(sp.encode([text], out_type=str), [pieces])
+++ self.assertEqual(sp.encode([text], out_type='proto'), [protos])
+++
++ detok_ids = self.sp_.DecodeIds(ids)
++ detok_pieces = self.sp_.DecodePieces(pieces)
++ self.assertEqual(sp.decode(ids), detok_ids)
++ self.assertEqual(sp.decode(pieces), detok_pieces)
+++ self.assertEqual(sp.decode([]), '')
+++ self.assertEqual(sp.decode([[]]), [''])
++
++ # add_bos, add_eos, reverse
++ self.assertEqual([sp.bos_id()] + ids, sp.encode(text, add_bos=True))
++ self.assertEqual(ids + [sp.eos_id()], sp.encode(text, add_eos=True))
+++ self.assertEqual(ids + [sp.eos_id()], sp.EncodeAsIds(text, add_eos=True))
++ rids = ids[:]
++ rids.reverse()
+++
++ self.assertEqual(rids, sp.encode(text, reverse=True))
+++ self.assertEqual(rids, sp.EncodeAsIds(text, reverse=True))
++
++ # different shape.
++ self.assertEqual([ids, ids2], sp.encode([text, text2]))
++@@ -351,6 +319,29 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual([text, text2], sp.decode([ids, ids2]))
++ self.assertEqual([text, text2], sp.decode([pieces, pieces2]))
++
+++ pieces = list(reversed(self.sp_.EncodeAsPieces(text)))
+++ self.assertEqual(pieces, sp.encode(text, reverse=True, out_type=str))
+++
+++ # emit unk piece
+++ unk_char = '藤'
+++ pieces = self.sp_.EncodeAsIds(unk_char, emit_unk_piece=True)
+++ pieces2 = self.sp_.encode(unk_char, out_type=int, emit_unk_piece=True)
+++ self.assertEqual(pieces[1], sp.unk_id())
+++ self.assertEqual(pieces2[1], sp.unk_id())
+++ self.assertEqual(pieces, pieces2)
+++
+++ pieces = self.sp_.EncodeAsPieces(unk_char, emit_unk_piece=True)
+++ pieces2 = self.sp_.encode(unk_char, out_type=str, emit_unk_piece=True)
+++ self.assertEqual(pieces[1], '<unk>')
+++ self.assertEqual(pieces2[1], '<unk>')
+++ self.assertEqual(pieces, pieces2)
+++
+++ pieces = self.sp_.EncodeAsPieces(unk_char, emit_unk_piece=False)
+++ pieces2 = self.sp_.encode(unk_char, out_type=str, emit_unk_piece=False)
+++ self.assertEqual(pieces[1], unk_char)
+++ self.assertEqual(pieces2[1], unk_char)
+++ self.assertEqual(pieces, pieces2)
+++
++ def test_new_api_init(self):
++ sp = spm.SentencePieceProcessor(
++ model_file=os.path.join('test', 'test_model.model'),
++@@ -361,7 +352,10 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ pieces = ['<s>'] + self.sp_.EncodeAsPieces(text) + ['</s>']
++ self.assertEqual(pieces, sp.encode(text))
++
++- def test_new_api_sampling(self):
+++ pieces = self.sp_.EncodeAsPieces(text) + ['</s>']
+++ self.assertEqual(pieces, sp.encode(text, add_bos=False, add_eos=True))
+++
+++ def test_sampling(self):
++ sp = spm.SentencePieceProcessor(
++ model_file=os.path.join('test', 'test_model.model'),
++ out_type=str,
++@@ -376,25 +370,35 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ ++ids2[' '.join(sp.encode('hello world', enable_sampling=False))]
++ self.assertEqual(len(ids2), 1)
++
++- def test_new_api_nbest(self):
+++ def test_nbest(self):
++ sp = spm.SentencePieceProcessor(
++ model_file=os.path.join('test', 'test_model.model'))
++- results = sp.nbest_encode('hello world', nbest_size=10, out_type=str)
+++ text = 'hello world'
+++ results = sp.nbest_encode(text, nbest_size=10, out_type=str)
+++ self.assertEqual(results, sp.NBestEncode(text, nbest_size=10, out_type=str))
++ for n in results:
++- self.assertEqual(sp.decode(n), 'hello world')
++- results = sp.nbest_encode('hello world', nbest_size=10, out_type=int)
+++ self.assertEqual(sp.decode(n), text)
+++ decoded = sp.decode(results)
+++ for n in decoded:
+++ self.assertEqual(n, text)
+++ results = sp.nbest_encode(text, nbest_size=10, out_type=int)
+++ self.assertEqual(results, sp.NBestEncode(text, nbest_size=10, out_type=int))
++ for n in results:
++- self.assertEqual(sp.decode(n), 'hello world')
+++ self.assertEqual(sp.decode(n), text)
+++ decoded = sp.decode(results)
+++ for n in decoded:
+++ self.assertEqual(n, text)
++
++- def test_new_api_sample_and_score(self):
+++ def test_sample_and_score(self):
++ sp = spm.SentencePieceProcessor(
++ model_file=os.path.join('test', 'test_model.model'))
++- results = sp.sample_encode_and_score('hello world', wor=True, out_type=str)
+++ text = 'hello world'
+++ results = sp.sample_encode_and_score(text, wor=True, out_type=str)
++ for n in results:
++- self.assertEqual(sp.decode(n[0]), 'hello world')
++- results = sp.sample_encode_and_score('hello world', wor=True, out_type=int)
+++ self.assertEqual(sp.decode(n[0]), text)
+++ results = sp.sample_encode_and_score(text, wor=True, out_type=int)
++ for n in results:
++- self.assertEqual(sp.decode(n[0]), 'hello world')
+++ self.assertEqual(sp.decode(n[0]), text)
++
++ def test_valid_range(self):
++ size = self.sp_.piece_size()
++@@ -412,6 +416,82 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ except:
++ self.assertTrue(True)
++
+++ def test_batch(self):
+++ sp = spm.SentencePieceProcessor(
+++ model_file=os.path.join('test', 'test_model.model'))
+++ with open(
+++ os.path.join(data_dir, 'botchan.txt'), 'r', encoding='utf-8') as file:
+++ texts = file.readlines()
+++
+++ r1 = sp.encode(texts, out_type=str, num_threads=None)
+++ r2 = sp.encode(texts, out_type=str, num_threads=1)
+++ r3 = sp.encode(texts, out_type=str, num_threads=-1)
+++ r4 = sp.encode(texts, out_type=str, num_threads=8)
+++ r5 = [sp.encode(s, out_type=str) for s in texts]
+++ self.assertEqual(r1, r2)
+++ self.assertEqual(r1, r3)
+++ self.assertEqual(r1, r4)
+++ self.assertEqual(r1, r5)
+++
+++ d1 = sp.decode(r1, num_threads=None)
+++ d2 = sp.decode(r2, num_threads=1)
+++ d3 = sp.decode(r3, num_threads=-1)
+++ d4 = sp.decode(r4, num_threads=8)
+++ d5 = [sp.decode(s) for s in r5]
+++ self.assertEqual(d1, d2)
+++ self.assertEqual(d1, d3)
+++ self.assertEqual(d1, d4)
+++ self.assertEqual(d1, d5)
+++
+++ r1 = sp.encode(texts, out_type=int, num_threads=None)
+++ r2 = sp.encode(texts, out_type=int, num_threads=1)
+++ r3 = sp.encode(texts, out_type=int, num_threads=-1)
+++ r4 = sp.encode(texts, out_type=int, num_threads=8)
+++ r5 = [sp.encode(s, out_type=int) for s in texts]
+++ self.assertEqual(r1, r2)
+++ self.assertEqual(r1, r3)
+++ self.assertEqual(r1, r4)
+++ self.assertEqual(r1, r5)
+++
+++ d1 = sp.decode(r1, num_threads=None)
+++ d2 = sp.decode(r2, num_threads=1)
+++ d3 = sp.decode(r3, num_threads=-1)
+++ d4 = sp.decode(r4, num_threads=8)
+++ d5 = [sp.decode(s) for s in r5]
+++ self.assertEqual(d1, d2)
+++ self.assertEqual(d1, d3)
+++ self.assertEqual(d1, d4)
+++ self.assertEqual(d1, d5)
+++
+++ r1 = sp.encode(texts, out_type='proto', num_threads=None)
+++ r2 = sp.encode(texts, out_type='proto', num_threads=1)
+++ r3 = sp.encode(texts, out_type='proto', num_threads=-1)
+++ r4 = sp.encode(texts, out_type='proto', num_threads=8)
+++ r5 = [sp.encode(s, out_type='proto') for s in texts]
+++ self.assertEqual(r1, r2)
+++ self.assertEqual(r1, r3)
+++ self.assertEqual(r1, r4)
+++ self.assertEqual(r1, r5)
+++
+++ e1 = sp.calculate_entropy(texts, theta=1.0, num_threads=10)
+++ e2 = sp.CalculateEntropy(texts, theta=1.0, num_threads=10)
+++ e3 = [sp.calculate_entropy(s, theta=1.0) for s in texts]
+++ self.assertEqual(e1, e2)
+++ self.assertEqual(e1, e3)
+++
+++ def test_pickle(self):
+++ with open('sp.pickle', 'wb') as f:
+++ pickle.dump(self.sp_, f)
+++
+++ id1 = self.sp_.encode('hello world.', out_type=int)
+++
+++ with open('sp.pickle', 'rb') as f:
+++ sp = pickle.load(f)
+++
+++ id2 = sp.encode('hello world.', out_type=int)
+++
+++ self.assertEqual(id1, id2)
+++
++
++ def suite():
++ suite = unittest.TestSuite()
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 8 Jun 2022 16:38:21 +0900
++Subject: remove debug symbols from wheel package
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/setup.py | 3 +++
++ 1 file changed, 3 insertions(+)
++
++diff --git a/python/setup.py b/python/setup.py
++index cfbf0db..198cba7 100755
++--- a/python/setup.py
+++++ b/python/setup.py
++@@ -93,6 +93,9 @@ class build_ext(_build_ext):
++ # See: https://github.com/neulab/xnmt/issues/199
++ if sys.platform == 'darwin':
++ cflags.append('-mmacosx-version-min=10.9')
+++ else:
+++ cflags.append('-Wl,-strip-all')
+++ libs.append('-Wl,-strip-all')
++ print('## cflags={}'.format(' '.join(cflags)))
++ print('## libs={}'.format(' '.join(libs)))
++ ext.extra_compile_args = cflags
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Mon, 13 Jun 2022 03:20:23 +0900
++Subject: allow tab character to be used in user_defined_symbols.
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ src/trainer_interface.cc | 11 ++++++++++-
++ src/util.cc | 5 ++---
++ 2 files changed, 12 insertions(+), 4 deletions(-)
++
++diff --git a/src/trainer_interface.cc b/src/trainer_interface.cc
++index ef0c370..5e26b75 100644
++--- a/src/trainer_interface.cc
+++++ b/src/trainer_interface.cc
++@@ -12,6 +12,8 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
+++#include "trainer_interface.h"
+++
++ #include <algorithm>
++ #include <cstdlib>
++ #include <memory>
++@@ -35,7 +37,6 @@
++ #include "third_party/absl/strings/str_format.h"
++ #include "third_party/absl/strings/str_join.h"
++ #include "third_party/absl/strings/str_split.h"
++-#include "trainer_interface.h"
++ #include "unicode_script.h"
++ #include "util.h"
++
++@@ -699,6 +700,14 @@ util::Status TrainerInterface::SaveVocab(absl::string_view filename) const {
++ auto output = filesystem::NewWritableFile(filename);
++ RETURN_IF_ERROR(output->status());
++
+++ for (const auto &piece : model_proto.pieces()) {
+++ if (piece.piece().find_first_of(" \t\r\n") != std::string::npos) {
+++ LOG(WARNING) << "The piece [" << piece.piece()
+++ << "] contains escaped characters that break the format of "
+++ << filename;
+++ }
+++ }
+++
++ if (trainer_spec_.vocabulary_output_piece_score()) {
++ for (const auto &piece : model_proto.pieces()) {
++ std::ostringstream os;
++diff --git a/src/util.cc b/src/util.cc
++index 8424448..8da16c4 100644
++--- a/src/util.cc
+++++ b/src/util.cc
++@@ -12,10 +12,10 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
++-#include <iostream>
++-
++ #include "util.h"
++
+++#include <iostream>
+++
++ namespace sentencepiece {
++
++ namespace {
++@@ -217,7 +217,6 @@ std::vector<std::string> StrSplitAsCSV(absl::string_view text) {
++
++ std::vector<std::string> result;
++ for (; str < eos; ++str) {
++- while (*str == ' ' || *str == '\t') ++str;
++ if (*str == '"') {
++ start = ++str;
++ end = start;
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Mon, 13 Jun 2022 16:46:18 +0900
++Subject: add test to use tab as user defined symbols..
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/test/sentencepiece_test.py | 11 ++++++-----
++ 1 file changed, 6 insertions(+), 5 deletions(-)
++
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index 99e36f3..6c48bcd 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -240,16 +240,18 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ input=[os.path.join(data_dir, 'botchan.txt')],
++ model_prefix='m',
++ vocab_size=1002,
++- user_defined_symbols=['foo', 'bar', ','],
+++ user_defined_symbols=['foo', 'bar', ',', ' ', '\t', '\b', '\n', '\r'],
++ logstream=open(os.devnull, 'w'))
++ sp = spm.SentencePieceProcessor()
++ sp.Load('m.model')
++- with open(
++- os.path.join(data_dir, 'botchan.txt'), 'r', encoding='utf-8') as file:
+++ with open(os.path.join(data_dir, 'botchan.txt'), 'r') as file:
++ for line in file:
++ sp.DecodePieces(sp.EncodeAsPieces(line))
++ sp.DecodeIds(sp.EncodeAsIds(line))
++
+++ s = 'hello\tworld\r\nthis\tis a \b pen'
+++ self.assertEqual(s, sp.decode(sp.encode(s)))
+++
++ def test_serialized_proto(self):
++ text = 'I saw a girl with a telescope.'
++ s1 = self.sp_.EncodeAsSerializedProto(text)
++@@ -419,8 +421,7 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ def test_batch(self):
++ sp = spm.SentencePieceProcessor(
++ model_file=os.path.join('test', 'test_model.model'))
++- with open(
++- os.path.join(data_dir, 'botchan.txt'), 'r', encoding='utf-8') as file:
+++ with open(os.path.join(data_dir, 'botchan.txt'), 'r') as file:
++ texts = file.readlines()
++
++ r1 = sp.encode(texts, out_type=str, num_threads=None)
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Tue, 14 Jun 2022 01:18:09 +0900
++Subject: Uses C++17 by default
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ CMakeLists.txt | 4 +-
++ python/setup.py | 6 +-
++ src/CMakeLists.txt | 1 -
++ src/sentencepiece_processor.h | 26 +-
++ third_party/absl/strings/string_view.cc | 267 -----------------
++ third_party/absl/strings/string_view.h | 508 --------------------------------
++ 6 files changed, 9 insertions(+), 803 deletions(-)
++ delete mode 100644 third_party/absl/strings/string_view.cc
++
++diff --git a/CMakeLists.txt b/CMakeLists.txt
++index a791f08..78379a3 100644
++--- a/CMakeLists.txt
+++++ b/CMakeLists.txt
++@@ -28,7 +28,7 @@ option(SPM_NO_THREADLOCAL "Disable thread_local operator" OFF)
++ option(SPM_USE_BUILTIN_PROTOBUF "Use built-in protobuf" ON)
++ option(SPM_USE_EXTERNAL_ABSL "Use external abseil" OFF)
++
++-set(CMAKE_CXX_STANDARD 11)
+++set(CMAKE_CXX_STANDARD 17)
++ set(CMAKE_CXX_STANDARD_REQUIRED ON)
++
++ if((CMAKE_CXX_COMPILER_ID STREQUAL "Clang" AND
++@@ -98,6 +98,8 @@ configure_file("${PROJECT_SOURCE_DIR}/config.h.in" "config.h")
++ configure_file("${PROJECT_SOURCE_DIR}/sentencepiece.pc.in" "sentencepiece.pc" @ONLY)
++
++ if (NOT MSVC)
+++ # suppress warning for C++11 features.
+++# add_definitions("-Wno-deprecated-declarations -Wno-deprecated-enum-enum-conversion")
++ install(FILES "${CMAKE_CURRENT_BINARY_DIR}/sentencepiece.pc" DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
++ endif()
++
++diff --git a/python/setup.py b/python/setup.py
++index 198cba7..fdf9394 100755
++--- a/python/setup.py
+++++ b/python/setup.py
++@@ -58,7 +58,7 @@ def is_sentencepiece_installed():
++
++
++ def get_cflags_and_libs(root):
++- cflags = ['-std=c++11', '-I' + os.path.join(root, 'include')]
+++ cflags = ['-std=c++17', '-I' + os.path.join(root, 'include')]
++ libs = []
++ if os.path.exists(os.path.join(root, 'lib/pkgconfig/sentencepiece.pc')):
++ libs = [
++@@ -109,13 +109,13 @@ if os.name == 'nt':
++ if sys.maxsize > 2**32:
++ arch = 'amd64'
++ if os.path.exists('..\\build\\root_{}\\lib'.format(arch)):
++- cflags = ['/MT', '/I..\\build\\root_{}\\include'.format(arch)]
+++ cflags = ['/std:c++17', '/MT', '/I..\\build\\root_{}\\include'.format(arch)]
++ libs = [
++ '..\\build\\root_{}\\lib\\sentencepiece.lib'.format(arch),
++ '..\\build\\root_{}\\lib\\sentencepiece_train.lib'.format(arch)
++ ]
++ else:
++- cflags = ['/MT', '/I..\\build\\root\\include']
+++ cflags = ['/std:c++17', '/MT', '/I..\\build\\root\\include']
++ libs = [
++ '..\\build\\root\\lib\\sentencepiece.lib',
++ '..\\build\\root\\lib\\sentencepiece_train.lib'
++diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
++index 8b7fb76..6cb3922 100644
++--- a/src/CMakeLists.txt
+++++ b/src/CMakeLists.txt
++@@ -25,7 +25,6 @@ if (SPM_USE_EXTERNAL_ABSL)
++ endif()
++ else()
++ set(ABSL_FLAGS_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/../third_party/absl/flags/flag.cc)
++- set(ABSL_STRINGS_SRCS ${CMAKE_CURRENT_SOURCE_DIR}/../third_party/absl/strings/string_view.cc)
++ endif()
++
++ if (SPM_USE_BUILTIN_PROTOBUF)
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index 3f9c20d..9d38214 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -18,33 +18,13 @@
++ #include <cstring>
++ #include <memory>
++ #include <string>
+++#include <string_view>
++ #include <utility>
++ #include <vector>
++
++-#if defined(_USE_INTERNAL_STRING_VIEW)
++-#include "third_party/absl/strings/string_view.h"
++-#elif defined(_USE_TF_STRING_VIEW)
++-#include "absl/strings/string_view.h"
++-#else
++-// Minimum absl::string_view class that is used only for
++-// the argument of public APIs.
++ namespace absl {
++-class string_view {
++- public:
++- string_view() : ptr_(nullptr), length_(0) {}
++- string_view(const std::string &str) : ptr_(str.data()), length_(str.size()) {}
++- string_view(const char *str) : ptr_(str), length_(std::strlen(str)) {}
++- string_view(const char *data, size_t len) : ptr_(data), length_(len) {}
++-
++- const char *data() const { return ptr_; }
++- size_t size() const { return length_; }
++-
++- private:
++- const char *ptr_ = nullptr;
++- size_t length_ = 0;
++-};
++-} // namespace absl
++-#endif
+++using std::string_view;
+++}
++
++ namespace sentencepiece {
++
++diff --git a/third_party/absl/strings/string_view.cc b/third_party/absl/strings/string_view.cc
++deleted file mode 100644
++index dce208d..0000000
++--- a/third_party/absl/strings/string_view.cc
+++++ /dev/null
++@@ -1,267 +0,0 @@
++-// Copyright 2017 The Abseil Authors.
++-//
++-// Licensed under the Apache License, Version 2.0 (the "License");
++-// you may not use this file except in compliance with the License.
++-// You may obtain a copy of the License at
++-//
++-// http://www.apache.org/licenses/LICENSE-2.0
++-//
++-// Unless required by applicable law or agreed to in writing, software
++-// distributed under the License is distributed on an "AS IS" BASIS,
++-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++-// See the License for the specific language governing permissions and
++-// limitations under the License.
++-
++-#include "third_party/absl/strings/string_view.h"
++-
++-#ifndef ABSL_HAVE_STD_STRING_VIEW
++-
++-#include <algorithm>
++-#include <climits>
++-#include <cstring>
++-#include <ostream>
++-
++-// #include "absl/strings/internal/memutil.h"
++-
++-namespace absl {
++-
++-namespace {
++-void WritePadding(std::ostream& o, size_t pad) {
++- char fill_buf[32];
++- memset(fill_buf, o.fill(), sizeof(fill_buf));
++- while (pad) {
++- size_t n = std::min(pad, sizeof(fill_buf));
++- o.write(fill_buf, n);
++- pad -= n;
++- }
++-}
++-
++-class LookupTable {
++- public:
++- // For each character in wanted, sets the index corresponding
++- // to the ASCII code of that character. This is used by
++- // the find_.*_of methods below to tell whether or not a character is in
++- // the lookup table in constant time.
++- explicit LookupTable(string_view wanted) {
++- for (char c : wanted) {
++- table_[Index(c)] = true;
++- }
++- }
++- bool operator[](char c) const { return table_[Index(c)]; }
++-
++- private:
++- static unsigned char Index(char c) { return static_cast<unsigned char>(c); }
++- bool table_[UCHAR_MAX + 1] = {};
++-};
++-
++-} // namespace
++-
++-std::ostream& operator<<(std::ostream& o, string_view piece) {
++- std::ostream::sentry sentry(o);
++- if (sentry) {
++- size_t lpad = 0;
++- size_t rpad = 0;
++- if (static_cast<size_t>(o.width()) > piece.size()) {
++- size_t pad = o.width() - piece.size();
++- if ((o.flags() & o.adjustfield) == o.left) {
++- rpad = pad;
++- } else {
++- lpad = pad;
++- }
++- }
++- if (lpad) WritePadding(o, lpad);
++- o.write(piece.data(), piece.size());
++- if (rpad) WritePadding(o, rpad);
++- o.width(0);
++- }
++- return o;
++-}
++-
++-string_view::size_type string_view::copy(char* buf, size_type n,
++- size_type pos) const {
++- size_type ulen = length_;
++- assert(pos <= ulen);
++- size_type rlen = std::min(ulen - pos, n);
++- if (rlen > 0) {
++- const char* start = ptr_ + pos;
++- std::copy(start, start + rlen, buf);
++- }
++- return rlen;
++-}
++-
++-namespace {
++-const char* memmatch(const char* phaystack, size_t haylen, const char* pneedle,
++- size_t neelen) {
++- if (0 == neelen) {
++- return phaystack; // even if haylen is 0
++- }
++- if (haylen < neelen) {
++- return nullptr;
++- }
++- const char* match;
++- const char* hayend = phaystack + haylen - neelen + 1;
++- while ((match = (const char*)(memchr(phaystack, pneedle[0],
++- hayend - phaystack)))) {
++- if (memcmp(match, pneedle, neelen) == 0) {
++- return match;
++- } else {
++- phaystack = match + 1;
++- }
++- }
++- return nullptr;
++-}
++-} // namespace
++-
++-string_view::size_type string_view::find(string_view s, size_type pos) const
++- noexcept {
++- if (empty() || pos > length_) {
++- if (empty() && pos == 0 && s.empty()) return 0;
++- return npos;
++- }
++- const char* result = memmatch(ptr_ + pos, length_ - pos, s.ptr_, s.length_);
++- return result ? result - ptr_ : npos;
++-}
++-
++-string_view::size_type string_view::find(char c, size_type pos) const noexcept {
++- if (empty() || pos >= length_) {
++- return npos;
++- }
++- const char* result =
++- static_cast<const char*>(memchr(ptr_ + pos, c, length_ - pos));
++- return result != nullptr ? result - ptr_ : npos;
++-}
++-
++-string_view::size_type string_view::rfind(string_view s, size_type pos) const
++- noexcept {
++- if (length_ < s.length_) return npos;
++- if (s.empty()) return std::min(length_, pos);
++- const char* last = ptr_ + std::min(length_ - s.length_, pos) + s.length_;
++- const char* result = std::find_end(ptr_, last, s.ptr_, s.ptr_ + s.length_);
++- return result != last ? result - ptr_ : npos;
++-}
++-
++-// Search range is [0..pos] inclusive. If pos == npos, search everything.
++-string_view::size_type string_view::rfind(char c, size_type pos) const
++- noexcept {
++- // Note: memrchr() is not available on Windows.
++- if (empty()) return npos;
++- for (size_type i = std::min(pos, length_ - 1);; --i) {
++- if (ptr_[i] == c) {
++- return i;
++- }
++- if (i == 0) break;
++- }
++- return npos;
++-}
++-
++-string_view::size_type string_view::find_first_of(string_view s,
++- size_type pos) const
++- noexcept {
++- if (empty() || s.empty()) {
++- return npos;
++- }
++- // Avoid the cost of LookupTable() for a single-character search.
++- if (s.length_ == 1) return find_first_of(s.ptr_[0], pos);
++- LookupTable tbl(s);
++- for (size_type i = pos; i < length_; ++i) {
++- if (tbl[ptr_[i]]) {
++- return i;
++- }
++- }
++- return npos;
++-}
++-
++-string_view::size_type string_view::find_first_not_of(string_view s,
++- size_type pos) const
++- noexcept {
++- if (empty()) return npos;
++- // Avoid the cost of LookupTable() for a single-character search.
++- if (s.length_ == 1) return find_first_not_of(s.ptr_[0], pos);
++- LookupTable tbl(s);
++- for (size_type i = pos; i < length_; ++i) {
++- if (!tbl[ptr_[i]]) {
++- return i;
++- }
++- }
++- return npos;
++-}
++-
++-string_view::size_type string_view::find_first_not_of(char c,
++- size_type pos) const
++- noexcept {
++- if (empty()) return npos;
++- for (; pos < length_; ++pos) {
++- if (ptr_[pos] != c) {
++- return pos;
++- }
++- }
++- return npos;
++-}
++-
++-string_view::size_type string_view::find_last_of(string_view s,
++- size_type pos) const noexcept {
++- if (empty() || s.empty()) return npos;
++- // Avoid the cost of LookupTable() for a single-character search.
++- if (s.length_ == 1) return find_last_of(s.ptr_[0], pos);
++- LookupTable tbl(s);
++- for (size_type i = std::min(pos, length_ - 1);; --i) {
++- if (tbl[ptr_[i]]) {
++- return i;
++- }
++- if (i == 0) break;
++- }
++- return npos;
++-}
++-
++-string_view::size_type string_view::find_last_not_of(string_view s,
++- size_type pos) const
++- noexcept {
++- if (empty()) return npos;
++- size_type i = std::min(pos, length_ - 1);
++- if (s.empty()) return i;
++- // Avoid the cost of LookupTable() for a single-character search.
++- if (s.length_ == 1) return find_last_not_of(s.ptr_[0], pos);
++- LookupTable tbl(s);
++- for (;; --i) {
++- if (!tbl[ptr_[i]]) {
++- return i;
++- }
++- if (i == 0) break;
++- }
++- return npos;
++-}
++-
++-string_view::size_type string_view::find_last_not_of(char c,
++- size_type pos) const
++- noexcept {
++- if (empty()) return npos;
++- size_type i = std::min(pos, length_ - 1);
++- for (;; --i) {
++- if (ptr_[i] != c) {
++- return i;
++- }
++- if (i == 0) break;
++- }
++- return npos;
++-}
++-
++-// MSVC has non-standard behavior that implicitly creates definitions for static
++-// const members. These implicit definitions conflict with explicit out-of-class
++-// member definitions that are required by the C++ standard, resulting in
++-// LNK1169 "multiply defined" errors at link time. __declspec(selectany) asks
++-// MSVC to choose only one definition for the symbol it decorates. See details
++-// at http://msdn.microsoft.com/en-us/library/34h23df8(v=vs.100).aspx
++-#ifdef _MSC_VER
++-#define ABSL_STRING_VIEW_SELECTANY __declspec(selectany)
++-#else
++-#define ABSL_STRING_VIEW_SELECTANY
++-#endif
++-
++-ABSL_STRING_VIEW_SELECTANY
++-constexpr string_view::size_type string_view::npos;
++-ABSL_STRING_VIEW_SELECTANY
++-constexpr string_view::size_type string_view::kMaxSize;
++-
++-} // namespace absl
++-
++-#endif // ABSL_HAVE_STD_STRING_VIEW
++diff --git a/third_party/absl/strings/string_view.h b/third_party/absl/strings/string_view.h
++index 68d46e3..9bb8b1c 100644
++--- a/third_party/absl/strings/string_view.h
+++++ b/third_party/absl/strings/string_view.h
++@@ -28,518 +28,10 @@
++ #define ABSL_STRINGS_STRING_VIEW_H_
++
++ #include <algorithm>
++-// #include "absl/base/config.h"
++-
++-#ifdef ABSL_HAVE_STD_STRING_VIEW
++-
++ #include <string_view>
++
++ namespace absl {
++ using std::string_view;
++-} // namespace absl
++-
++-#else // ABSL_HAVE_STD_STRING_VIEW
++-
++-#include <cassert>
++-#include <cstddef>
++-#include <cstring>
++-#include <iosfwd>
++-#include <iterator>
++-#include <limits>
++-#include <string>
++-
++-#ifdef __has_builtin
++-#define ABSL_HAVE_BUILTIN(x) __has_builtin(x)
++-#else
++-#define ABSL_HAVE_BUILTIN(x) 0
++-#endif
++-
++-// #include "absl/base/internal/throw_delegate.h"
++-// #include "absl/base/macros.h"
++-// #include "absl/base/port.h"
++-
++-namespace absl {
++-
++-// absl::string_view
++-//
++-// A `string_view` provides a lightweight view into the std::string data
++-// provided by a `std::string`, double-quoted std::string literal, character
++-// array, or even another `string_view`. A `string_view` does *not* own the
++-// std::string to which it points, and that data cannot be modified through the
++-// view.
++-//
++-// You can use `string_view` as a function or method parameter anywhere a
++-// parameter can receive a double-quoted std::string literal, `const char*`,
++-// `std::string`, or another `absl::string_view` argument with no need to copy
++-// the std::string data. Systematic use of `string_view` within function
++-// arguments reduces data copies and `strlen()` calls.
++-//
++-// Because of its small size, prefer passing `string_view` by value:
++-//
++-// void MyFunction(absl::string_view arg);
++-//
++-// If circumstances require, you may also pass one by const reference:
++-//
++-// void MyFunction(const absl::string_view& arg); // not preferred
++-//
++-// Passing by value generates slightly smaller code for many architectures.
++-//
++-// In either case, the source data of the `string_view` must outlive the
++-// `string_view` itself.
++-//
++-// A `string_view` is also suitable for local variables if you know that the
++-// lifetime of the underlying object is longer than the lifetime of your
++-// `string_view` variable. However, beware of binding a `string_view` to a
++-// temporary value:
++-//
++-// // BAD use of string_view: lifetime problem
++-// absl::string_view sv = obj.ReturnAString();
++-//
++-// // GOOD use of string_view: str outlives sv
++-// std::string str = obj.ReturnAString();
++-// absl::string_view sv = str;
++-//
++-// Due to lifetime issues, a `string_view` is sometimes a poor choice for a
++-// return value and usually a poor choice for a data member. If you do use a
++-// `string_view` this way, it is your responsibility to ensure that the object
++-// pointed to by the `string_view` outlives the `string_view`.
++-//
++-// A `string_view` may represent a whole std::string or just part of a
++-// std::string. For example, when splitting a std::string,
++-// `std::vector<absl::string_view>` is a natural data type for the output.
++-//
++-//
++-// When constructed from a source which is nul-terminated, the `string_view`
++-// itself will not include the nul-terminator unless a specific size (including
++-// the nul) is passed to the constructor. As a result, common idioms that work
++-// on nul-terminated strings do not work on `string_view` objects. If you write
++-// code that scans a `string_view`, you must check its length rather than test
++-// for nul, for example. Note, however, that nuls may still be embedded within
++-// a `string_view` explicitly.
++-//
++-// You may create a null `string_view` in two ways:
++-//
++-// absl::string_view sv();
++-// absl::string_view sv(nullptr, 0);
++-//
++-// For the above, `sv.data() == nullptr`, `sv.length() == 0`, and
++-// `sv.empty() == true`. Also, if you create a `string_view` with a non-null
++-// pointer then `sv.data() != nullptr`. Thus, you can use `string_view()` to
++-// signal an undefined value that is different from other `string_view` values
++-// in a similar fashion to how `const char* p1 = nullptr;` is different from
++-// `const char* p2 = "";`. However, in practice, it is not recommended to rely
++-// on this behavior.
++-//
++-// Be careful not to confuse a null `string_view` with an empty one. A null
++-// `string_view` is an empty `string_view`, but some empty `string_view`s are
++-// not null. Prefer checking for emptiness over checking for null.
++-//
++-// There are many ways to create an empty string_view:
++-//
++-// const char* nullcp = nullptr;
++-// // string_view.size() will return 0 in all cases.
++-// absl::string_view();
++-// absl::string_view(nullcp, 0);
++-// absl::string_view("");
++-// absl::string_view("", 0);
++-// absl::string_view("abcdef", 0);
++-// absl::string_view("abcdef" + 6, 0);
++-//
++-// All empty `string_view` objects whether null or not, are equal:
++-//
++-// absl::string_view() == absl::string_view("", 0)
++-// absl::string_view(nullptr, 0) == absl:: string_view("abcdef"+6, 0)
++-class string_view {
++- public:
++- using traits_type = std::char_traits<char>;
++- using value_type = char;
++- using pointer = char*;
++- using const_pointer = const char*;
++- using reference = char&;
++- using const_reference = const char&;
++- using const_iterator = const char*;
++- using iterator = const_iterator;
++- using const_reverse_iterator = std::reverse_iterator<const_iterator>;
++- using reverse_iterator = const_reverse_iterator;
++- using size_type = size_t;
++- using difference_type = std::ptrdiff_t;
++-
++- static constexpr size_type npos = static_cast<size_type>(-1);
++-
++- // Null `string_view` constructor
++- constexpr string_view() noexcept : ptr_(nullptr), length_(0) {}
++-
++- // Implicit constructors
++-
++- template <typename Allocator>
++- string_view( // NOLINT(runtime/explicit)
++- const std::basic_string<char, std::char_traits<char>, Allocator>&
++- str) noexcept
++- : ptr_(str.data()), length_(CheckLengthInternal(str.size())) {}
++-
++- // Implicit constructor of a `string_view` from nul-terminated `str`. When
++- // accepting possibly null strings, use `absl::NullSafeStringView(str)`
++- // instead (see below).
++- constexpr string_view(const char* str) // NOLINT(runtime/explicit)
++- : ptr_(str), length_(CheckLengthInternal(StrLenInternal(str))) {}
++-
++- // Implicit constructor of a `string_view` from a `const char*` and length.
++- constexpr string_view(const char* data, size_type len)
++- : ptr_(data), length_(CheckLengthInternal(len)) {}
++-
++- // NOTE: Harmlessly omitted to work around gdb bug.
++- // constexpr string_view(const string_view&) noexcept = default;
++- // string_view& operator=(const string_view&) noexcept = default;
++-
++- // Iterators
++-
++- // string_view::begin()
++- //
++- // Returns an iterator pointing to the first character at the beginning of the
++- // `string_view`, or `end()` if the `string_view` is empty.
++- constexpr const_iterator begin() const noexcept { return ptr_; }
++-
++- // string_view::end()
++- //
++- // Returns an iterator pointing just beyond the last character at the end of
++- // the `string_view`. This iterator acts as a placeholder; attempting to
++- // access it results in undefined behavior.
++- constexpr const_iterator end() const noexcept { return ptr_ + length_; }
++-
++- // string_view::cbegin()
++- //
++- // Returns a const iterator pointing to the first character at the beginning
++- // of the `string_view`, or `end()` if the `string_view` is empty.
++- constexpr const_iterator cbegin() const noexcept { return begin(); }
++-
++- // string_view::cend()
++- //
++- // Returns a const iterator pointing just beyond the last character at the end
++- // of the `string_view`. This pointer acts as a placeholder; attempting to
++- // access its element results in undefined behavior.
++- constexpr const_iterator cend() const noexcept { return end(); }
++-
++- // string_view::rbegin()
++- //
++- // Returns a reverse iterator pointing to the last character at the end of the
++- // `string_view`, or `rend()` if the `string_view` is empty.
++- const_reverse_iterator rbegin() const noexcept {
++- return const_reverse_iterator(end());
++- }
++-
++- // string_view::rend()
++- //
++- // Returns a reverse iterator pointing just before the first character at the
++- // beginning of the `string_view`. This pointer acts as a placeholder;
++- // attempting to access its element results in undefined behavior.
++- const_reverse_iterator rend() const noexcept {
++- return const_reverse_iterator(begin());
++- }
++-
++- // string_view::crbegin()
++- //
++- // Returns a const reverse iterator pointing to the last character at the end
++- // of the `string_view`, or `crend()` if the `string_view` is empty.
++- const_reverse_iterator crbegin() const noexcept { return rbegin(); }
++-
++- // string_view::crend()
++- //
++- // Returns a const reverse iterator pointing just before the first character
++- // at the beginning of the `string_view`. This pointer acts as a placeholder;
++- // attempting to access its element results in undefined behavior.
++- const_reverse_iterator crend() const noexcept { return rend(); }
++-
++- // Capacity Utilities
++-
++- // string_view::size()
++- //
++- // Returns the number of characters in the `string_view`.
++- constexpr size_type size() const noexcept { return length_; }
++-
++- // string_view::length()
++- //
++- // Returns the number of characters in the `string_view`. Alias for `size()`.
++- constexpr size_type length() const noexcept { return size(); }
++-
++- // string_view::max_size()
++- //
++- // Returns the maximum number of characters the `string_view` can hold.
++- constexpr size_type max_size() const noexcept { return kMaxSize; }
++-
++- // string_view::empty()
++- //
++- // Checks if the `string_view` is empty (refers to no characters).
++- constexpr bool empty() const noexcept { return length_ == 0; }
++-
++- // std::string:view::operator[]
++- //
++- // Returns the ith element of an `string_view` using the array operator.
++- // Note that this operator does not perform any bounds checking.
++- constexpr const_reference operator[](size_type i) const { return ptr_[i]; }
++-
++- // string_view::front()
++- //
++- // Returns the first element of a `string_view`.
++- constexpr const_reference front() const { return ptr_[0]; }
++-
++- // string_view::back()
++- //
++- // Returns the last element of a `string_view`.
++- constexpr const_reference back() const { return ptr_[size() - 1]; }
++-
++- // string_view::data()
++- //
++- // Returns a pointer to the underlying character array (which is of course
++- // stored elsewhere). Note that `string_view::data()` may contain embedded nul
++- // characters, but the returned buffer may or may not be nul-terminated;
++- // therefore, do not pass `data()` to a routine that expects a nul-terminated
++- // std::string.
++- constexpr const_pointer data() const noexcept { return ptr_; }
++-
++- // Modifiers
++-
++- // string_view::remove_prefix()
++- //
++- // Removes the first `n` characters from the `string_view`. Note that the
++- // underlying std::string is not changed, only the view.
++- void remove_prefix(size_type n) {
++- assert(n <= length_);
++- ptr_ += n;
++- length_ -= n;
++- }
++-
++- // string_view::remove_suffix()
++- //
++- // Removes the last `n` characters from the `string_view`. Note that the
++- // underlying std::string is not changed, only the view.
++- void remove_suffix(size_type n) {
++- assert(n <= length_);
++- length_ -= n;
++- }
++-
++- // string_view::swap()
++- //
++- // Swaps this `string_view` with another `string_view`.
++- void swap(string_view& s) noexcept {
++- auto t = *this;
++- *this = s;
++- s = t;
++- }
++-
++- // Explicit conversion operators
++-
++- // Converts to `std::basic_string`.
++- template <typename A>
++- explicit operator std::basic_string<char, traits_type, A>() const {
++- if (!data()) return {};
++- return std::basic_string<char, traits_type, A>(data(), size());
++- }
++-
++- // string_view::copy()
++- //
++- // Copies the contents of the `string_view` at offset `pos` and length `n`
++- // into `buf`.
++- size_type copy(char* buf, size_type n, size_type pos = 0) const;
++-
++- // string_view::substr()
++- //
++- // Returns a "substring" of the `string_view` (at offset `pos` and length
++- // `n`) as another string_view. This function throws `std::out_of_bounds` if
++- // `pos > size'.
++- string_view substr(size_type pos, size_type n = npos) const {
++- n = std::min(n, length_ - pos);
++- return string_view(ptr_ + pos, n);
++- }
++-
++- // string_view::compare()
++- //
++- // Performs a lexicographical comparison between the `string_view` and
++- // another `absl::string_view), returning -1 if `this` is less than, 0 if
++- // `this` is equal to, and 1 if `this` is greater than the passed std::string
++- // view. Note that in the case of data equality, a further comparison is made
++- // on the respective sizes of the two `string_view`s to determine which is
++- // smaller, equal, or greater.
++- int compare(string_view x) const noexcept {
++- auto min_length = std::min(length_, x.length_);
++- if (min_length > 0) {
++- int r = memcmp(ptr_, x.ptr_, min_length);
++- if (r < 0) return -1;
++- if (r > 0) return 1;
++- }
++- if (length_ < x.length_) return -1;
++- if (length_ > x.length_) return 1;
++- return 0;
++- }
++-
++- // Overload of `string_view::compare()` for comparing a substring of the
++- // 'string_view` and another `absl::string_view`.
++- int compare(size_type pos1, size_type count1, string_view v) const {
++- return substr(pos1, count1).compare(v);
++- }
++-
++- // Overload of `string_view::compare()` for comparing a substring of the
++- // `string_view` and a substring of another `absl::string_view`.
++- int compare(size_type pos1, size_type count1, string_view v, size_type pos2,
++- size_type count2) const {
++- return substr(pos1, count1).compare(v.substr(pos2, count2));
++- }
++-
++- // Overload of `string_view::compare()` for comparing a `string_view` and a
++- // a different C-style std::string `s`.
++- int compare(const char* s) const { return compare(string_view(s)); }
++-
++- // Overload of `string_view::compare()` for comparing a substring of the
++- // `string_view` and a different std::string C-style std::string `s`.
++- int compare(size_type pos1, size_type count1, const char* s) const {
++- return substr(pos1, count1).compare(string_view(s));
++- }
++-
++- // Overload of `string_view::compare()` for comparing a substring of the
++- // `string_view` and a substring of a different C-style std::string `s`.
++- int compare(size_type pos1, size_type count1, const char* s,
++- size_type count2) const {
++- return substr(pos1, count1).compare(string_view(s, count2));
++- }
++-
++- // Find Utilities
++-
++- // string_view::find()
++- //
++- // Finds the first occurrence of the substring `s` within the `string_view`,
++- // returning the position of the first character's match, or `npos` if no
++- // match was found.
++- size_type find(string_view s, size_type pos = 0) const noexcept;
++-
++- // Overload of `string_view::find()` for finding the given character `c`
++- // within the `string_view`.
++- size_type find(char c, size_type pos = 0) const noexcept;
++-
++- // string_view::rfind()
++- //
++- // Finds the last occurrence of a substring `s` within the `string_view`,
++- // returning the position of the first character's match, or `npos` if no
++- // match was found.
++- size_type rfind(string_view s, size_type pos = npos) const noexcept;
++-
++- // Overload of `string_view::rfind()` for finding the last given character `c`
++- // within the `string_view`.
++- size_type rfind(char c, size_type pos = npos) const noexcept;
++-
++- // string_view::find_first_of()
++- //
++- // Finds the first occurrence of any of the characters in `s` within the
++- // `string_view`, returning the start position of the match, or `npos` if no
++- // match was found.
++- size_type find_first_of(string_view s, size_type pos = 0) const noexcept;
++-
++- // Overload of `string_view::find_first_of()` for finding a character `c`
++- // within the `string_view`.
++- size_type find_first_of(char c, size_type pos = 0) const noexcept {
++- return find(c, pos);
++- }
++-
++- // string_view::find_last_of()
++- //
++- // Finds the last occurrence of any of the characters in `s` within the
++- // `string_view`, returning the start position of the match, or `npos` if no
++- // match was found.
++- size_type find_last_of(string_view s, size_type pos = npos) const noexcept;
++-
++- // Overload of `string_view::find_last_of()` for finding a character `c`
++- // within the `string_view`.
++- size_type find_last_of(char c, size_type pos = npos) const noexcept {
++- return rfind(c, pos);
++- }
++-
++- // string_view::find_first_not_of()
++- //
++- // Finds the first occurrence of any of the characters not in `s` within the
++- // `string_view`, returning the start position of the first non-match, or
++- // `npos` if no non-match was found.
++- size_type find_first_not_of(string_view s, size_type pos = 0) const noexcept;
++-
++- // Overload of `string_view::find_first_not_of()` for finding a character
++- // that is not `c` within the `string_view`.
++- size_type find_first_not_of(char c, size_type pos = 0) const noexcept;
++-
++- // string_view::find_last_not_of()
++- //
++- // Finds the last occurrence of any of the characters not in `s` within the
++- // `string_view`, returning the start position of the last non-match, or
++- // `npos` if no non-match was found.
++- size_type find_last_not_of(string_view s,
++- size_type pos = npos) const noexcept;
++-
++- // Overload of `string_view::find_last_not_of()` for finding a character
++- // that is not `c` within the `string_view`.
++- size_type find_last_not_of(char c, size_type pos = npos) const noexcept;
++-
++- private:
++- static constexpr size_type kMaxSize =
++- std::numeric_limits<difference_type>::max();
++-
++- // check whether __builtin_strlen is provided by the compiler.
++- // GCC doesn't have __has_builtin()
++- // (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66970),
++- // but has __builtin_strlen according to
++- // https://gcc.gnu.org/onlinedocs/gcc-4.7.0/gcc/Other-Builtins.html.
++-#if ABSL_HAVE_BUILTIN(__builtin_strlen) || \
++- (defined(__GNUC__) && !defined(__clang__))
++- static constexpr size_type StrLenInternal(const char* str) {
++- return str ? __builtin_strlen(str) : 0;
++- }
++-#else
++- static constexpr size_type StrLenInternal(const char* str) {
++- return str ? strlen(str) : 0;
++- }
++-#endif
++-
++- static constexpr size_type CheckLengthInternal(size_type len) { return len; }
++-
++- const char* ptr_;
++- size_type length_;
++-};
++-
++-// This large function is defined inline so that in a fairly common case where
++-// one of the arguments is a literal, the compiler can elide a lot of the
++-// following comparisons.
++-inline bool operator==(string_view x, string_view y) noexcept {
++- auto len = x.size();
++- if (len != y.size()) {
++- return false;
++- }
++- return x.data() == y.data() || len <= 0 ||
++- memcmp(x.data(), y.data(), len) == 0;
++-}
++-
++-inline bool operator!=(string_view x, string_view y) noexcept {
++- return !(x == y);
++-}
++-
++-inline bool operator<(string_view x, string_view y) noexcept {
++- auto min_size = std::min(x.size(), y.size());
++- const int r = min_size == 0 ? 0 : memcmp(x.data(), y.data(), min_size);
++- return (r < 0) || (r == 0 && x.size() < y.size());
++-}
++-
++-inline bool operator>(string_view x, string_view y) noexcept { return y < x; }
++-
++-inline bool operator<=(string_view x, string_view y) noexcept {
++- return !(y < x);
++-}
++-
++-inline bool operator>=(string_view x, string_view y) noexcept {
++- return !(x < y);
++-}
++-
++-// IO Insertion Operator
++-std::ostream& operator<<(std::ostream& o, string_view piece);
++-
++-} // namespace absl
++-
++-#endif // ABSL_HAVE_STD_STRING_VIEW
++-
++-namespace absl {
++
++ // ClippedSubstr()
++ //
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Tue, 14 Jun 2022 02:00:43 +0900
++Subject: Uses std::atomic to define global variable
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ src/common.h | 13 -------------
++ src/util.cc | 13 +++++++------
++ 2 files changed, 7 insertions(+), 19 deletions(-)
++
++diff --git a/src/common.h b/src/common.h
++index 7595634..6ec4c09 100644
++--- a/src/common.h
+++++ b/src/common.h
++@@ -50,19 +50,6 @@ typedef uint32_t char32;
++ typedef uint32_t uint32;
++ typedef uint64_t uint64;
++
++-static constexpr uint8 kuint8max = ((uint8)0xFF);
++-static constexpr uint16 kuint16max = ((uint16)0xFFFF);
++-static constexpr uint32 kuint32max = ((uint32)0xFFFFFFFF);
++-static constexpr uint64 kuint64max = ((uint64)(0xFFFFFFFFFFFFFFFF));
++-static constexpr int8 kint8min = ((int8)~0x7F);
++-static constexpr int8 kint8max = ((int8)0x7F);
++-static constexpr int16 kint16min = ((int16)~0x7FFF);
++-static constexpr int16 kint16max = ((int16)0x7FFF);
++-static constexpr int32 kint32min = ((int32)~0x7FFFFFFF);
++-static constexpr int32 kint32max = ((int32)0x7FFFFFFF);
++-static constexpr int64 kint64min = ((int64)(~0x7FFFFFFFFFFFFFFF));
++-static constexpr int64 kint64max = ((int64)(0x7FFFFFFFFFFFFFFF));
++-
++ static constexpr uint32 kUnicodeError = 0xFFFD;
++
++ #if defined(OS_WIN) && defined(UNICODE) && defined(_UNICODE)
++diff --git a/src/util.cc b/src/util.cc
++index 8da16c4..f99c73a 100644
++--- a/src/util.cc
+++++ b/src/util.cc
++@@ -14,27 +14,28 @@
++
++ #include "util.h"
++
+++#include <atomic>
++ #include <iostream>
++
++ namespace sentencepiece {
++
++ namespace {
++ constexpr unsigned int kDefaultSeed = static_cast<unsigned int>(-1);
++-static unsigned int g_seed = kDefaultSeed;
++-static int g_minloglevel = 0;
+++static std::atomic<unsigned int> g_seed = kDefaultSeed;
+++static std::atomic<int> g_minloglevel = 0;
++ } // namespace
++
++ void SetRandomGeneratorSeed(unsigned int seed) {
++- if (seed != kDefaultSeed) g_seed = seed;
+++ if (seed != kDefaultSeed) g_seed.store(seed);
++ }
++
++ uint32 GetRandomGeneratorSeed() {
++- return g_seed == kDefaultSeed ? std::random_device{}() : g_seed;
+++ return g_seed == kDefaultSeed ? std::random_device{}() : g_seed.load();
++ }
++
++ namespace logging {
++-int GetMinLogLevel() { return g_minloglevel; }
++-void SetMinLogLevel(int v) { g_minloglevel = v; }
+++int GetMinLogLevel() { return g_minloglevel.load(); }
+++void SetMinLogLevel(int v) { g_minloglevel.store(v); }
++ } // namespace logging
++
++ namespace string_util {
--- /dev/null
--- /dev/null
++From: Kentaro Hayashi <kenhys@gmail.com>
++Date: Tue, 14 Jun 2022 20:40:59 +0900
++Subject: Fix a typo
++
++gu rantees ->
++guarantees
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ src/trainer_interface.cc | 4 ++--
++ 1 file changed, 2 insertions(+), 2 deletions(-)
++
++diff --git a/src/trainer_interface.cc b/src/trainer_interface.cc
++index 5e26b75..7270f29 100644
++--- a/src/trainer_interface.cc
+++++ b/src/trainer_interface.cc
++@@ -460,11 +460,11 @@ END:
++ }
++ if (trainer_spec_.differential_privacy_noise_level() <= 0) {
++ LOG(WARNING) << "Private version with <=0 noise level will give "
++- "infinity epsilon gurantees.";
+++ "infinity epsilon guarantees.";
++ }
++ if (trainer_spec_.differential_privacy_clipping_threshold() <= 0) {
++ LOG(WARNING) << "Private version with <=0 clipping threshold will give "
++- "infinity epsilon gurantees.";
+++ "infinity epsilon guarantees.";
++ }
++
++ // Add noise to all the sentences via threadpool.
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 15 Jun 2022 01:29:55 +0900
++Subject: Uses absl::string_view as much as possible
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/__init__.py | 4 +-
++ python/src/sentencepiece/sentencepiece.i | 92 ++-----
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 329 ++++++++++++++++--------
++ src/builder.cc | 2 +-
++ src/builder.h | 2 +-
++ src/common.h | 3 +-
++ src/error.cc | 9 +-
++ src/sentencepiece_processor.cc | 36 ++-
++ src/sentencepiece_processor.h | 28 +-
++ src/sentencepiece_trainer.h | 8 +-
++ src/spec_parser.h | 16 +-
++ src/spm_encode_main.cc | 22 +-
++ src/util.cc | 27 +-
++ 13 files changed, 333 insertions(+), 245 deletions(-)
++
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index cba3b70..1543d32 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -93,8 +93,8 @@ class SentencePieceProcessor(object):
++ def SampleEncodeAndScoreAsIds(self, input, num_samples, theta, wor, include_best):
++ return _sentencepiece.SentencePieceProcessor_SampleEncodeAndScoreAsIds(self, input, num_samples, theta, wor, include_best)
++
++- def CalculateEntropy(self, text, theta):
++- return _sentencepiece.SentencePieceProcessor_CalculateEntropy(self, text, theta)
+++ def CalculateEntropy(self, *args):
+++ return _sentencepiece.SentencePieceProcessor_CalculateEntropy(self, *args)
++
++ def GetPieceSize(self):
++ return _sentencepiece.SentencePieceProcessor_GetPieceSize(self)
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 3a822bc..40373ce 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -37,6 +37,7 @@ class PyInputString {
++ str_ = nullptr;
++ }
++ }
+++ absl::string_view str() const { return absl::string_view(data(), size()); }
++ const char* data() const { return str_; }
++ Py_ssize_t size() const { return size_; }
++ bool IsAvalable() const { return str_ != nullptr; }
++@@ -179,7 +180,7 @@ inline void CheckIds(const std::vector<int> &ids, int num_pieces) {
++ }
++ }
++
++-inline void CheckIds(const std::vector<std::string> &ids, int num_pieces) {}
+++inline void CheckIds(const std::vector<absl::string_view> &ids, int num_pieces) {}
++
++ class ThreadPool {
++ public:
++@@ -266,6 +267,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ %ignore sentencepiece::util::Status;
++ %ignore sentencepiece::util::StatusCode;
++ %ignore absl::string_view;
+++%ignore std::string_view;
++ %ignore sentencepiece::SentencePieceText;
++ %ignore sentencepiece::NormalizerSpec;
++ %ignore sentencepiece::TrainerSpec;
++@@ -386,7 +388,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return $self->DecodeIds(ids);
++ }
++
++- std::string _DecodePieces(const std::vector<std::string> &pieces) const {
+++ std::string _DecodePieces(const std::vector<absl::string_view> &pieces) const {
++ return $self->DecodePieces(pieces);
++ }
++
++@@ -397,7 +399,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++
++ sentencepiece::util::bytes _DecodePiecesAsSerializedProto(
++- const std::vector<std::string> &pieces) const {
+++ const std::vector<absl::string_view> &pieces) const {
++ CheckIds(pieces, $self->GetPieceSize());
++ return $self->DecodePiecesAsSerializedProto(pieces);
++ }
++@@ -416,12 +418,12 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++
++ std::vector<std::string> _DecodePiecesBatch(
++- const std::vector<std::vector<std::string>> &ins, int num_threads) const {
+++ const std::vector<std::vector<absl::string_view>> &ins, int num_threads) const {
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePieces, std::string, std::string);
++ }
++
++ BytesArray _DecodePiecesAsSerializedProtoBatch(
++- const std::vector<std::vector<std::string>> &ins, int num_threads) const {
+++ const std::vector<std::vector<absl::string_view>> &ins, int num_threads) const {
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsSerializedProto, std::string,
++ sentencepiece::util::bytes);
++ }
++@@ -1029,14 +1031,14 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ %typemap(out) std::vector<int> {
++ $result = PyList_New($1.size());
++ for (size_t i = 0; i < $1.size(); ++i) {
++- PyList_SetItem($result, i, PyInt_FromLong(static_cast<long>($1[i])));
+++ PyList_SET_ITEM($result, i, PyInt_FromLong(static_cast<long>($1[i])));
++ }
++ }
++
++ %typemap(out) std::vector<float> {
++ $result = PyList_New($1.size());
++ for (size_t i = 0; i < $1.size(); ++i) {
++- PyList_SetItem($result, i, PyFloat_FromDouble(static_cast<double>($1[i])));
+++ PyList_SET_ITEM($result, i, PyFloat_FromDouble(static_cast<double>($1[i])));
++ }
++ }
++
++@@ -1045,9 +1047,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (size_t i = 0; i < $1.size(); ++i) {
++ PyObject *obj = PyList_New($1[i].size());
++ for (size_t j = 0; j < $1[i].size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>($1[i][j])));
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>($1[i][j])));
++ }
++- PyList_SetItem($result, i, obj);
+++ PyList_SET_ITEM($result, i, obj);
++ }
++ }
++
++@@ -1055,14 +1057,14 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ PyObject *input_type = resultobj;
++ $result = PyList_New($1.size());
++ for (size_t i = 0; i < $1.size(); ++i) {
++- PyList_SetItem($result, i, MakePyOutputString($1[i], input_type));
+++ PyList_SET_ITEM($result, i, MakePyOutputString($1[i], input_type));
++ }
++ }
++
++ %typemap(out) BytesArray {
++ $result = PyList_New($1.size());
++ for (size_t i = 0; i < $1.size(); ++i) {
++- PyList_SetItem($result, i, MakePyOutputBytes($1[i]));
+++ PyList_SET_ITEM($result, i, MakePyOutputBytes($1[i]));
++ }
++ }
++
++@@ -1072,9 +1074,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (size_t i = 0; i < $1.size(); ++i) {
++ PyObject *obj = PyList_New($1[i].size());
++ for (size_t j = 0; j < $1[i].size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString($1[i][j], input_type));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString($1[i][j], input_type));
++ }
++- PyList_SetItem($result, i, obj);
+++ PyList_SET_ITEM($result, i, obj);
++ }
++ }
++
++@@ -1118,51 +1120,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- $1 = absl::string_view(ustring.data(), ustring.size());
++-}
++-
++-%typemap(in) const std::vector<std::string>& {
++- std::vector<std::string> *out = nullptr;
++- if (PyList_Check($input)) {
++- const size_t size = PyList_Size($input);
++- out = new std::vector<std::string>(size);
++- for (size_t i = 0; i < size; ++i) {
++- const PyInputString ustring(PyList_GetItem($input, i));
++- if (ustring.IsAvalable()) {
++- (*out)[i].assign(ustring.data(), ustring.size());
++- } else {
++- PyErr_SetString(PyExc_TypeError, "list must contain strings");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- }
++- } else {
++- PyErr_SetString(PyExc_TypeError, "not a list");
++- SWIG_fail;
++- }
++- $1 = out;
++-}
++-
++-%typemap(in) const std::vector<absl::string_view>& {
++- std::vector<absl::string_view> *out = nullptr;
++- if (PyList_Check($input)) {
++- const size_t size = PyList_Size($input);
++- out = new std::vector<std::string>(size);
++- for (size_t i = 0; i < size; ++i) {
++- const PyInputString ustring(PyList_GetItem($input, i));
++- if (ustring.IsAvalable()) {
++- (*out)[i] = absl::string_view(ustring.data(), ustring.size());
++- } else {
++- PyErr_SetString(PyExc_TypeError, "list must contain strings");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- }
++- } else {
++- PyErr_SetString(PyExc_TypeError, "not a list");
++- SWIG_fail;
++- }
++- $1 = out;
+++ $1 = ustring.str();
++ }
++
++ %typemap(in) const std::vector<absl::string_view>& {
++@@ -1173,7 +1131,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem($input, i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -1208,11 +1166,11 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ $1 = out;
++ }
++
++-%typemap(in) const std::vector<std::vector<std::string>>& {
++- std::vector<std::vector<std::string>> *out = nullptr;
+++%typemap(in) const std::vector<std::vector<absl::string_view>>& {
+++ std::vector<std::vector<absl::string_view>> *out = nullptr;
++ if (PyList_Check($input)) {
++ const size_t size = PyList_Size($input);
++- out = new std::vector<std::vector<std::string>>(size);
+++ out = new std::vector<std::vector<absl::string_view>>(size);
++ for (size_t i = 0; i < size; ++i) {
++ PyObject *o = PyList_GetItem($input, i);
++ if (PyList_Check(o)) {
++@@ -1221,7 +1179,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (size_t j = 0; j < size2; ++j) {
++ const PyInputString ustring(PyList_GetItem(o, j));
++ if (ustring.IsAvalable()) {
++- (*out)[i][j].assign(ustring.data(), ustring.size());
+++ (*out)[i][j] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
++ SWIG_fail;
++@@ -1302,9 +1260,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (size_t i = 0; i < $1.size(); ++i) {
++ PyObject *obj = PyList_New($1[i].first.size());
++ for (size_t j = 0; j < $1[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString($1[i].first[j], input_type));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString($1[i].first[j], input_type));
++ }
++- PyList_SetItem($result, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>($1[i].second))));
+++ PyList_SET_ITEM($result, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>($1[i].second))));
++ }
++ }
++
++@@ -1313,9 +1271,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (size_t i = 0; i < $1.size(); ++i) {
++ PyObject *obj = PyList_New($1[i].first.size());
++ for (size_t j = 0; j < $1[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>($1[i].first[j])));
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>($1[i].first[j])));
++ }
++- PyList_SetItem($result, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>($1[i].second))));
+++ PyList_SET_ITEM($result, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>($1[i].second))));
++ }
++ }
++
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 6df3880..36ce38c 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -2693,16 +2693,16 @@ SWIGINTERN PyObject *SWIG_PyStaticMethod_New(PyObject *SWIGUNUSEDPARM(self), PyO
++ /* -------- TYPES TABLE (BEGIN) -------- */
++
++ #define SWIGTYPE_p_char swig_types[0]
++-#define SWIGTYPE_p_sentencepiece__SentenceIterator swig_types[1]
++-#define SWIGTYPE_p_sentencepiece__SentencePieceProcessor swig_types[2]
++-#define SWIGTYPE_p_sentencepiece__SentencePieceTrainer swig_types[3]
++-#define SWIGTYPE_p_std__string swig_types[4]
++-#define SWIGTYPE_p_std__unordered_mapT_std__string_std__string_t swig_types[5]
++-#define SWIGTYPE_p_std__vectorT_absl__string_view_t swig_types[6]
++-#define SWIGTYPE_p_std__vectorT_int_t swig_types[7]
++-#define SWIGTYPE_p_std__vectorT_std__string_t swig_types[8]
++-#define SWIGTYPE_p_std__vectorT_std__vectorT_int_t_t swig_types[9]
++-#define SWIGTYPE_p_std__vectorT_std__vectorT_std__string_t_t swig_types[10]
+++#define SWIGTYPE_p_float swig_types[1]
+++#define SWIGTYPE_p_sentencepiece__SentenceIterator swig_types[2]
+++#define SWIGTYPE_p_sentencepiece__SentencePieceProcessor swig_types[3]
+++#define SWIGTYPE_p_sentencepiece__SentencePieceTrainer swig_types[4]
+++#define SWIGTYPE_p_std__string swig_types[5]
+++#define SWIGTYPE_p_std__unordered_mapT_std__string_std__string_t swig_types[6]
+++#define SWIGTYPE_p_std__vectorT_absl__string_view_t swig_types[7]
+++#define SWIGTYPE_p_std__vectorT_int_t swig_types[8]
+++#define SWIGTYPE_p_std__vectorT_std__vectorT_absl__string_view_t_t swig_types[9]
+++#define SWIGTYPE_p_std__vectorT_std__vectorT_int_t_t swig_types[10]
++ static swig_type_info *swig_types[12];
++ static swig_module_info swig_module = {swig_types, 11, 0, 0, 0, 0};
++ #define SWIG_TypeQuery(name) SWIG_TypeQueryModule(&swig_module, &swig_module, name)
++@@ -2843,6 +2843,7 @@ class PyInputString {
++ str_ = nullptr;
++ }
++ }
+++ absl::string_view str() const { return absl::string_view(data(), size()); }
++ const char* data() const { return str_; }
++ Py_ssize_t size() const { return size_; }
++ bool IsAvalable() const { return str_ != nullptr; }
++@@ -2985,7 +2986,7 @@ inline void CheckIds(const std::vector<int> &ids, int num_pieces) {
++ }
++ }
++
++-inline void CheckIds(const std::vector<std::string> &ids, int num_pieces) {}
+++inline void CheckIds(const std::vector<absl::string_view> &ids, int num_pieces) {}
++
++ class ThreadPool {
++ public:
++@@ -3473,14 +3474,14 @@ SWIGINTERN std::string sentencepiece_SentencePieceProcessor__DecodeIds(sentencep
++ CheckIds(ids, self->GetPieceSize());
++ return self->DecodeIds(ids);
++ }
++-SWIGINTERN std::string sentencepiece_SentencePieceProcessor__DecodePieces(sentencepiece::SentencePieceProcessor const *self,std::vector< std::string > const &pieces){
+++SWIGINTERN std::string sentencepiece_SentencePieceProcessor__DecodePieces(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &pieces){
++ return self->DecodePieces(pieces);
++ }
++ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__DecodeIdsAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
++ CheckIds(ids, self->GetPieceSize());
++ return self->DecodeIdsAsSerializedProto(ids);
++ }
++-SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,std::vector< std::string > const &pieces){
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &pieces){
++ CheckIds(pieces, self->GetPieceSize());
++ return self->DecodePiecesAsSerializedProto(pieces);
++ }
++@@ -3491,10 +3492,10 @@ SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodeIdsAsSerialize
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIdsAsSerializedProto, int,
++ sentencepiece::util::bytes);
++ }
++-SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodePiecesBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< std::string > > const &ins,int num_threads){
+++SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodePiecesBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< absl::string_view > > const &ins,int num_threads){
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePieces, std::string, std::string);
++ }
++-SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< std::string > > const &ins,int num_threads){
+++SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< absl::string_view > > const &ins,int num_threads){
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsSerializedProto, std::string,
++ sentencepiece::util::bytes);
++ }
++@@ -3718,7 +3719,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_LoadFromSerializedProto(PyObje
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ {
++ try {
++@@ -3763,7 +3764,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SetEncodeExtraOptions(PyObject
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ {
++ try {
++@@ -3808,7 +3809,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SetDecodeExtraOptions(PyObject
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ {
++ try {
++@@ -3834,7 +3835,7 @@ fail:
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SetVocabulary(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::string > *arg2 = 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++ PyObject *swig_obj[2] ;
++@@ -3847,14 +3848,14 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SetVocabulary(PyObject *SWIGUN
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<std::string> *out = nullptr;
+++ std::vector<absl::string_view> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::string>(size);
+++ out = new std::vector<absl::string_view>(size);
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i].assign(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -3869,7 +3870,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SetVocabulary(PyObject *SWIGUN
++ }
++ {
++ try {
++- result = (arg1)->SetVocabulary((std::vector< std::string > const &)*arg2);
+++ result = (arg1)->SetVocabulary((std::vector< absl::string_view > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -3955,7 +3956,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_LoadVocabulary(PyObject *SWIGU
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -3983,6 +3984,66 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy__SWIG_0(PyObject *SWIGUNUSEDPARM(self), Py_ssize_t nobjs, PyObject **swig_obj) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ absl::string_view arg2 ;
+++ float arg3 ;
+++ float *arg4 = (float *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ float val3 ;
+++ int ecode3 = 0 ;
+++ void *argp4 = 0 ;
+++ int res4 = 0 ;
+++ sentencepiece::util::Status result;
+++
+++ if ((nobjs < 4) || (nobjs > 4)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = ustring.str();
+++ }
+++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "3"" of type '" "float""'");
+++ }
+++ arg3 = static_cast< float >(val3);
+++ res4 = SWIG_ConvertPtr(swig_obj[3], &argp4,SWIGTYPE_p_float, 0 | 0 );
+++ if (!SWIG_IsOK(res4)) {
+++ SWIG_exception_fail(SWIG_ArgError(res4), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "4"" of type '" "float *""'");
+++ }
+++ arg4 = reinterpret_cast< float * >(argp4);
+++ {
+++ try {
+++ result = ((sentencepiece::SentencePieceProcessor const *)arg1)->CalculateEntropy(arg2,arg3,arg4);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ if (!(&result)->ok()) {
+++ SWIG_exception(ToSwigError((&result)->code()), (&result)->ToString().c_str());
+++ }
+++ resultobj = SWIG_From_bool((&result)->ok());
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++@@ -4017,7 +4078,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces(P
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -4054,9 +4115,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces(P
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].first.size());
++ for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i].first[j], input_type));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString(result[i].first[j], input_type));
++ }
++- PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
+++ PyList_SET_ITEM(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++ }
++ }
++ return resultobj;
++@@ -4099,7 +4160,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyOb
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -4135,9 +4196,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyOb
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].first.size());
++ for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
++ }
++- PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
+++ PyList_SET_ITEM(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++ }
++ }
++ return resultobj;
++@@ -4146,7 +4207,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy__SWIG_1(PyObject *SWIGUNUSEDPARM(self), Py_ssize_t nobjs, PyObject **swig_obj) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ absl::string_view arg2 ;
++@@ -4155,10 +4216,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy(PyObject *SWI
++ int res1 = 0 ;
++ float val3 ;
++ int ecode3 = 0 ;
++- PyObject *swig_obj[3] ;
++ float result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_CalculateEntropy", 3, 3, swig_obj)) SWIG_fail;
+++ if ((nobjs < 3) || (nobjs > 3)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++@@ -4171,7 +4231,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy(PyObject *SWI
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -4194,6 +4254,67 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy(PyObject *self, PyObject *args) {
+++ Py_ssize_t argc;
+++ PyObject *argv[5] = {
+++ 0
+++ };
+++
+++ if (!(argc = SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_CalculateEntropy", 0, 4, argv))) SWIG_fail;
+++ --argc;
+++ if (argc == 3) {
+++ int _v;
+++ void *vptr = 0;
+++ int res = SWIG_ConvertPtr(argv[0], &vptr, SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0);
+++ _v = SWIG_CheckState(res);
+++ if (_v) {
+++ int res = SWIG_AsCharPtrAndSize(argv[1], 0, NULL, 0);
+++ _v = SWIG_CheckState(res);
+++ if (_v) {
+++ {
+++ int res = SWIG_AsVal_float(argv[2], NULL);
+++ _v = SWIG_CheckState(res);
+++ }
+++ if (_v) {
+++ return _wrap_SentencePieceProcessor_CalculateEntropy__SWIG_1(self, argc, argv);
+++ }
+++ }
+++ }
+++ }
+++ if (argc == 4) {
+++ int _v;
+++ void *vptr = 0;
+++ int res = SWIG_ConvertPtr(argv[0], &vptr, SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0);
+++ _v = SWIG_CheckState(res);
+++ if (_v) {
+++ int res = SWIG_AsCharPtrAndSize(argv[1], 0, NULL, 0);
+++ _v = SWIG_CheckState(res);
+++ if (_v) {
+++ {
+++ int res = SWIG_AsVal_float(argv[2], NULL);
+++ _v = SWIG_CheckState(res);
+++ }
+++ if (_v) {
+++ void *vptr = 0;
+++ int res = SWIG_ConvertPtr(argv[3], &vptr, SWIGTYPE_p_float, 0);
+++ _v = SWIG_CheckState(res);
+++ if (_v) {
+++ return _wrap_SentencePieceProcessor_CalculateEntropy__SWIG_0(self, argc, argv);
+++ }
+++ }
+++ }
+++ }
+++ }
+++
+++fail:
+++ SWIG_Python_RaiseOrModifyTypeError("Wrong number or type of arguments for overloaded function 'SentencePieceProcessor_CalculateEntropy'.\n"
+++ " Possible C/C++ prototypes are:\n"
+++ " sentencepiece::SentencePieceProcessor::CalculateEntropy(absl::string_view,float,float *) const\n"
+++ " sentencepiece::SentencePieceProcessor::CalculateEntropy(absl::string_view,float) const\n");
+++ return 0;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_GetPieceSize(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++@@ -4247,7 +4368,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_PieceToId(PyObject *SWIGUNUSED
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ {
++ try {
++@@ -4675,7 +4796,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_LoadFromFile(PyObject *SWIGUNU
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ {
++ try {
++@@ -4741,7 +4862,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIds(PyObject *SWIGUNU
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -4790,7 +4911,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIds(PyObject *SWIGUNU
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, PyInt_FromLong(static_cast<long>(result[i])));
+++ PyList_SET_ITEM(resultobj, i, PyInt_FromLong(static_cast<long>(result[i])));
++ }
++ }
++ return resultobj;
++@@ -4842,7 +4963,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPieces(PyObject *SWIG
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -4892,7 +5013,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPieces(PyObject *SWIG
++ PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputString(result[i], input_type));
++ }
++ }
++ return resultobj;
++@@ -4944,7 +5065,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProto(PyObj
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -5046,7 +5167,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SW
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -5113,9 +5234,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SW
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].size());
++ for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
++ }
++- PyList_SetItem(resultobj, i, obj);
+++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++ }
++ {
++@@ -5177,7 +5298,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -5245,9 +5366,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].size());
++ for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i][j], input_type));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString(result[i][j], input_type));
++ }
++- PyList_SetItem(resultobj, i, obj);
+++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++ }
++ {
++@@ -5309,7 +5430,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -5374,7 +5495,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputBytes(result[i]));
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
++ }
++ }
++ {
++@@ -5452,7 +5573,7 @@ fail:
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::string > *arg2 = 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++ PyObject *swig_obj[2] ;
++@@ -5465,14 +5586,14 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePieces(PyObject *SWIGUN
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<std::string> *out = nullptr;
+++ std::vector<absl::string_view> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::string>(size);
+++ out = new std::vector<absl::string_view>(size);
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i].assign(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -5487,7 +5608,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePieces(PyObject *SWIGUN
++ }
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__DecodePieces((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::string > const &)*arg2);
+++ result = sentencepiece_SentencePieceProcessor__DecodePieces((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5572,7 +5693,7 @@ fail:
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::string > *arg2 = 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++ PyObject *swig_obj[2] ;
++@@ -5585,14 +5706,14 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<std::string> *out = nullptr;
+++ std::vector<absl::string_view> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::string>(size);
+++ out = new std::vector<absl::string_view>(size);
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i].assign(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -5607,7 +5728,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ }
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::string > const &)*arg2);
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5695,7 +5816,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsBatch(PyObject *SWIG
++ PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputString(result[i], input_type));
++ }
++ }
++ {
++@@ -5775,7 +5896,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsAsSerializedProtoBat
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputBytes(result[i]));
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
++ }
++ }
++ {
++@@ -5793,7 +5914,7 @@ fail:
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::vector< std::string > > *arg2 = 0 ;
+++ std::vector< std::vector< absl::string_view > > *arg2 = 0 ;
++ int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++@@ -5809,10 +5930,10 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *S
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<std::vector<std::string>> *out = nullptr;
+++ std::vector<std::vector<absl::string_view>> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::vector<std::string>>(size);
+++ out = new std::vector<std::vector<absl::string_view>>(size);
++ for (size_t i = 0; i < size; ++i) {
++ PyObject *o = PyList_GetItem(swig_obj[1], i);
++ if (PyList_Check(o)) {
++@@ -5821,7 +5942,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *S
++ for (size_t j = 0; j < size2; ++j) {
++ const PyInputString ustring(PyList_GetItem(o, j));
++ if (ustring.IsAvalable()) {
++- (*out)[i][j].assign(ustring.data(), ustring.size());
+++ (*out)[i][j] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
++ SWIG_fail;
++@@ -5846,7 +5967,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *S
++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__DecodePiecesBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< std::string > > const &)*arg2,arg3);
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< absl::string_view > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5857,17 +5978,11 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *S
++ PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputString(result[i], input_type));
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputString(result[i], input_type));
++ }
++ }
++- {
++- delete arg2;
++- }
++ return resultobj;
++ fail:
++- {
++- delete arg2;
++- }
++ return NULL;
++ }
++
++@@ -5875,7 +5990,7 @@ fail:
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- std::vector< std::vector< std::string > > *arg2 = 0 ;
+++ std::vector< std::vector< absl::string_view > > *arg2 = 0 ;
++ int arg3 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++@@ -5891,10 +6006,10 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++- std::vector<std::vector<std::string>> *out = nullptr;
+++ std::vector<std::vector<absl::string_view>> *out = nullptr;
++ if (PyList_Check(swig_obj[1])) {
++ const size_t size = PyList_Size(swig_obj[1]);
++- out = new std::vector<std::vector<std::string>>(size);
+++ out = new std::vector<std::vector<absl::string_view>>(size);
++ for (size_t i = 0; i < size; ++i) {
++ PyObject *o = PyList_GetItem(swig_obj[1], i);
++ if (PyList_Check(o)) {
++@@ -5903,7 +6018,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ for (size_t j = 0; j < size2; ++j) {
++ const PyInputString ustring(PyList_GetItem(o, j));
++ if (ustring.IsAvalable()) {
++- (*out)[i][j].assign(ustring.data(), ustring.size());
+++ (*out)[i][j] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
++ SWIG_fail;
++@@ -5928,7 +6043,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< std::string > > const &)*arg2,arg3);
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< absl::string_view > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5938,17 +6053,11 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, MakePyOutputBytes(result[i]));
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
++ }
++ }
++- {
++- delete arg2;
++- }
++ return resultobj;
++ fail:
++- {
++- delete arg2;
++- }
++ return NULL;
++ }
++
++@@ -5990,7 +6099,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SW
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -6031,9 +6140,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsIds(PyObject *SW
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].size());
++ for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
++ }
++- PyList_SetItem(resultobj, i, obj);
+++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++ }
++ return resultobj;
++@@ -6079,7 +6188,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -6121,9 +6230,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsPieces(PyObject
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].size());
++ for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i][j], input_type));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString(result[i][j], input_type));
++ }
++- PyList_SetItem(resultobj, i, obj);
+++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++ }
++ return resultobj;
++@@ -6169,7 +6278,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsSerializedProto(
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -6260,7 +6369,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds(PyO
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -6316,9 +6425,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds(PyO
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].first.size());
++ for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
++ }
++- PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
+++ PyList_SET_ITEM(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++ }
++ }
++ return resultobj;
++@@ -6373,7 +6482,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -6430,9 +6539,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].first.size());
++ for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SetItem(obj, j, MakePyOutputString(result[i].first[j], input_type));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString(result[i].first[j], input_type));
++ }
++- PyList_SetItem(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
+++ PyList_SET_ITEM(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++ }
++ }
++ return resultobj;
++@@ -6466,7 +6575,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__CalculateEntropy(PyObject *SW
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg2 = absl::string_view(ustring.data(), ustring.size());
+++ arg2 = ustring.str();
++ }
++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++@@ -6518,7 +6627,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__CalculateEntropyBatch(PyObjec
++ for (size_t i = 0; i < size; ++i) {
++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
++ if (ustring.IsAvalable()) {
++- (*out)[i] = absl::string_view(ustring.data(), ustring.size());
+++ (*out)[i] = ustring.str();
++ } else {
++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
++ SWIG_fail;
++@@ -6553,7 +6662,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__CalculateEntropyBatch(PyObjec
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SetItem(resultobj, i, PyFloat_FromDouble(static_cast<double>(result[i])));
+++ PyList_SET_ITEM(resultobj, i, PyFloat_FromDouble(static_cast<double>(result[i])));
++ }
++ }
++ {
++@@ -6623,7 +6732,7 @@ SWIGINTERN PyObject *_wrap_SentencePieceTrainer__TrainFromString(PyObject *SWIGU
++ SWIG_fail;
++ }
++ resultobj = ustring.input_type();
++- arg1 = absl::string_view(ustring.data(), ustring.size());
+++ arg1 = ustring.str();
++ }
++ {
++ try {
++@@ -6966,6 +7075,7 @@ static PyMethodDef SwigMethods_proxydocs[] = {
++ /* -------- TYPE CONVERSION AND EQUIVALENCE RULES (BEGIN) -------- */
++
++ static swig_type_info _swigt__p_char = {"_p_char", "char *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_float = {"_p_float", "float *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_sentencepiece__SentenceIterator = {"_p_sentencepiece__SentenceIterator", "sentencepiece::SentenceIterator *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_sentencepiece__SentencePieceProcessor = {"_p_sentencepiece__SentencePieceProcessor", "sentencepiece::SentencePieceProcessor *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_sentencepiece__SentencePieceTrainer = {"_p_sentencepiece__SentencePieceTrainer", "sentencepiece::SentencePieceTrainer *", 0, 0, (void*)0, 0};
++@@ -6973,12 +7083,12 @@ static swig_type_info _swigt__p_std__string = {"_p_std__string", "sentencepiece:
++ static swig_type_info _swigt__p_std__unordered_mapT_std__string_std__string_t = {"_p_std__unordered_mapT_std__string_std__string_t", "std::unordered_map< std::string,std::string > *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__vectorT_absl__string_view_t = {"_p_std__vectorT_absl__string_view_t", "std::vector< absl::string_view > *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__vectorT_int_t = {"_p_std__vectorT_int_t", "std::vector< int > *", 0, 0, (void*)0, 0};
++-static swig_type_info _swigt__p_std__vectorT_std__string_t = {"_p_std__vectorT_std__string_t", "std::vector< std::string > *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_std__vectorT_std__vectorT_absl__string_view_t_t = {"_p_std__vectorT_std__vectorT_absl__string_view_t_t", "std::vector< std::vector< absl::string_view > > *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_std__vectorT_std__vectorT_int_t_t = {"_p_std__vectorT_std__vectorT_int_t_t", "std::vector< std::vector< int > > *", 0, 0, (void*)0, 0};
++-static swig_type_info _swigt__p_std__vectorT_std__vectorT_std__string_t_t = {"_p_std__vectorT_std__vectorT_std__string_t_t", "std::vector< std::vector< std::string > > *", 0, 0, (void*)0, 0};
++
++ static swig_type_info *swig_type_initial[] = {
++ &_swigt__p_char,
+++ &_swigt__p_float,
++ &_swigt__p_sentencepiece__SentenceIterator,
++ &_swigt__p_sentencepiece__SentencePieceProcessor,
++ &_swigt__p_sentencepiece__SentencePieceTrainer,
++@@ -6986,12 +7096,12 @@ static swig_type_info *swig_type_initial[] = {
++ &_swigt__p_std__unordered_mapT_std__string_std__string_t,
++ &_swigt__p_std__vectorT_absl__string_view_t,
++ &_swigt__p_std__vectorT_int_t,
++- &_swigt__p_std__vectorT_std__string_t,
+++ &_swigt__p_std__vectorT_std__vectorT_absl__string_view_t_t,
++ &_swigt__p_std__vectorT_std__vectorT_int_t_t,
++- &_swigt__p_std__vectorT_std__vectorT_std__string_t_t,
++ };
++
++ static swig_cast_info _swigc__p_char[] = { {&_swigt__p_char, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_float[] = { {&_swigt__p_float, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_sentencepiece__SentenceIterator[] = { {&_swigt__p_sentencepiece__SentenceIterator, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_sentencepiece__SentencePieceProcessor[] = { {&_swigt__p_sentencepiece__SentencePieceProcessor, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_sentencepiece__SentencePieceTrainer[] = { {&_swigt__p_sentencepiece__SentencePieceTrainer, 0, 0, 0},{0, 0, 0, 0}};
++@@ -6999,12 +7109,12 @@ static swig_cast_info _swigc__p_std__string[] = { {&_swigt__p_std__string, 0, 0
++ static swig_cast_info _swigc__p_std__unordered_mapT_std__string_std__string_t[] = { {&_swigt__p_std__unordered_mapT_std__string_std__string_t, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__vectorT_absl__string_view_t[] = { {&_swigt__p_std__vectorT_absl__string_view_t, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__vectorT_int_t[] = { {&_swigt__p_std__vectorT_int_t, 0, 0, 0},{0, 0, 0, 0}};
++-static swig_cast_info _swigc__p_std__vectorT_std__string_t[] = { {&_swigt__p_std__vectorT_std__string_t, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_std__vectorT_std__vectorT_absl__string_view_t_t[] = { {&_swigt__p_std__vectorT_std__vectorT_absl__string_view_t_t, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_std__vectorT_std__vectorT_int_t_t[] = { {&_swigt__p_std__vectorT_std__vectorT_int_t_t, 0, 0, 0},{0, 0, 0, 0}};
++-static swig_cast_info _swigc__p_std__vectorT_std__vectorT_std__string_t_t[] = { {&_swigt__p_std__vectorT_std__vectorT_std__string_t_t, 0, 0, 0},{0, 0, 0, 0}};
++
++ static swig_cast_info *swig_cast_initial[] = {
++ _swigc__p_char,
+++ _swigc__p_float,
++ _swigc__p_sentencepiece__SentenceIterator,
++ _swigc__p_sentencepiece__SentencePieceProcessor,
++ _swigc__p_sentencepiece__SentencePieceTrainer,
++@@ -7012,9 +7122,8 @@ static swig_cast_info *swig_cast_initial[] = {
++ _swigc__p_std__unordered_mapT_std__string_std__string_t,
++ _swigc__p_std__vectorT_absl__string_view_t,
++ _swigc__p_std__vectorT_int_t,
++- _swigc__p_std__vectorT_std__string_t,
+++ _swigc__p_std__vectorT_std__vectorT_absl__string_view_t_t,
++ _swigc__p_std__vectorT_std__vectorT_int_t_t,
++- _swigc__p_std__vectorT_std__vectorT_std__string_t_t,
++ };
++
++
++diff --git a/src/builder.cc b/src/builder.cc
++index 0fc7f24..822f6fc 100644
++--- a/src/builder.cc
+++++ b/src/builder.cc
++@@ -272,7 +272,7 @@ util::Status Builder::DecompileCharsMap(absl::string_view blob,
++ }
++
++ // static
++-util::Status Builder::GetPrecompiledCharsMap(const std::string &name,
+++util::Status Builder::GetPrecompiledCharsMap(absl::string_view name,
++ std::string *output) {
++ CHECK_OR_RETURN(output);
++
++diff --git a/src/builder.h b/src/builder.h
++index 95c5168..094da72 100644
++--- a/src/builder.h
+++++ b/src/builder.h
++@@ -51,7 +51,7 @@ class Builder {
++ CharsMap *chars_map);
++
++ // Returns a pre-compiled binary index with `name`.
++- static util::Status GetPrecompiledCharsMap(const std::string &name,
+++ static util::Status GetPrecompiledCharsMap(absl::string_view name,
++ std::string *output);
++
++ // Makes a normalization mapping based on NFKC.
++diff --git a/src/common.h b/src/common.h
++index 6ec4c09..ab07d85 100644
++--- a/src/common.h
+++++ b/src/common.h
++@@ -71,8 +71,7 @@ char (&ArraySizeHelper(const T (&array)[N]))[N];
++ namespace sentencepiece {
++ #ifdef OS_WIN
++ namespace win32 {
++-std::wstring Utf8ToWide(const std::string &input);
++-std::string WideToUtf8(const std::wstring &input);
+++std::wstring Utf8ToWide(const absl::string_view input);
++ } // namespace win32
++ #endif
++
++diff --git a/src/error.cc b/src/error.cc
++index a226d98..10faa2d 100644
++--- a/src/error.cc
+++++ b/src/error.cc
++@@ -61,15 +61,10 @@ struct Status::Rep {
++ std::string error_message;
++ };
++
++-Status::Status(StatusCode code, const char* error_message) : rep_(new Rep) {
++- rep_->code = code;
++- rep_->error_message = error_message;
++-}
++-
++-Status::Status(StatusCode code, const std::string& error_message)
+++Status::Status(StatusCode code, absl::string_view error_message)
++ : rep_(new Rep) {
++ rep_->code = code;
++- rep_->error_message = error_message;
+++ rep_->error_message = std::string(error_message);
++ }
++
++ Status::Status(const Status& s)
++diff --git a/src/sentencepiece_processor.cc b/src/sentencepiece_processor.cc
++index 4d697be..331fc90 100644
++--- a/src/sentencepiece_processor.cc
+++++ b/src/sentencepiece_processor.cc
++@@ -48,6 +48,12 @@ const char kDefaultUnknownSymbol[] = " \xE2\x81\x87 ";
++
++ // REPLACEMENT CHARACTER (U+FFFD) in UTF-8.
++ const char kReplacementCharacter[] = "\xef\xbf\xbd";
+++
+++std::vector<absl::string_view> ToPieceArray(const std::vector<std::string> &v) {
+++ std::vector<absl::string_view> out(v.size());
+++ for (int i = 0; i < v.size(); ++i) out[i] = v[i];
+++ return out;
+++}
++ } // namespace
++
++ SentencePieceProcessor::SentencePieceProcessor() {}
++@@ -146,7 +152,7 @@ util::Status SentencePieceProcessor::status() const {
++ }
++
++ util::Status SentencePieceProcessor::SetVocabulary(
++- const std::vector<std::string> &valid_vocab) {
+++ const std::vector<absl::string_view> &valid_vocab) {
++ RETURN_IF_ERROR(status());
++
++ // TODO(taku): supports vocabulary constraint in BPE model.
++@@ -154,7 +160,8 @@ util::Status SentencePieceProcessor::SetVocabulary(
++ CHECK_OR_RETURN(type == TrainerSpec::UNIGRAM || type == TrainerSpec::BPE)
++ << "Vocabulary constraint is only enabled in subword units.";
++
++- const std::set<std::string> vocab(valid_vocab.begin(), valid_vocab.end());
+++ const std::set<absl::string_view> vocab(valid_vocab.begin(),
+++ valid_vocab.end());
++
++ for (int i = 0; i < model_proto_->pieces_size(); ++i) {
++ auto *piece = model_proto_->mutable_pieces(i);
++@@ -207,7 +214,7 @@ util::Status SentencePieceProcessor::LoadVocabulary(absl::string_view filename,
++ }
++ }
++
++- return SetVocabulary(vocab);
+++ return SetVocabulary(ToPieceArray(vocab));
++ }
++
++ #define CHECK_OR_RETURN_STATUS_STL(container) \
++@@ -250,6 +257,12 @@ util::Status SentencePieceProcessor::Encode(absl::string_view input,
++
++ util::Status SentencePieceProcessor::Decode(
++ const std::vector<std::string> &pieces, std::string *detokenized) const {
+++ return Decode(ToPieceArray(pieces), detokenized);
+++}
+++
+++util::Status SentencePieceProcessor::Decode(
+++ const std::vector<absl::string_view> &pieces,
+++ std::string *detokenized) const {
++ CHECK_OR_RETURN_STATUS_STL(detokenized);
++
++ SentencePieceText spt;
++@@ -593,6 +606,12 @@ util::Status SentencePieceProcessor::CalculateEntropy(absl::string_view input,
++
++ util::Status SentencePieceProcessor::Decode(
++ const std::vector<std::string> &pieces, SentencePieceText *spt) const {
+++ return Decode(ToPieceArray(pieces), spt);
+++}
+++
+++util::Status SentencePieceProcessor::Decode(
+++ const std::vector<absl::string_view> &pieces,
+++ SentencePieceText *spt) const {
++ CHECK_OR_RETURN_STATUS_PROTO(spt);
++
++ const char *unk_surface = kDefaultUnknownSymbol;
++@@ -637,9 +656,9 @@ util::Status SentencePieceProcessor::Decode(
++ has_bos_ws);
++ };
++
++- for (const std::string &w : pieces) {
+++ for (absl::string_view w : pieces) {
++ auto *sp = spt->add_pieces();
++- sp->set_piece(w);
+++ sp->mutable_piece()->assign(w.data(), w.size());
++ sp->set_id(PieceToId(w));
++ }
++
++@@ -779,6 +798,13 @@ std::string SentencePieceProcessor::DecodePiecesAsSerializedProto(
++ return spt.SerializeAsString();
++ }
++
+++std::string SentencePieceProcessor::DecodePiecesAsSerializedProto(
+++ const std::vector<absl::string_view> &pieces) const {
+++ SentencePieceText spt;
+++ if (!Decode(pieces, &spt).ok()) return "";
+++ return spt.SerializeAsString();
+++}
+++
++ std::string SentencePieceProcessor::DecodeIdsAsSerializedProto(
++ const std::vector<int> &ids) const {
++ SentencePieceText spt;
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index 9d38214..8c72656 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -22,9 +22,11 @@
++ #include <utility>
++ #include <vector>
++
+++#ifndef SWIG
++ namespace absl {
++ using std::string_view;
++ }
+++#endif // SWIG
++
++ namespace sentencepiece {
++
++@@ -58,8 +60,7 @@ class Status {
++ public:
++ Status();
++ ~Status();
++- Status(StatusCode code, const char *error_message);
++- Status(StatusCode code, const std::string &error_message);
+++ Status(StatusCode code, absl::string_view error_message);
++ Status(const Status &s);
++ void operator=(const Status &s);
++ bool operator==(const Status &s) const;
++@@ -204,7 +205,7 @@ class SentencePieceProcessor {
++ // Restricts the vocabulary set.
++ // The input sentences are encoded into the tokens in `valid_vocab`.
++ virtual util::Status SetVocabulary(
++- const std::vector<std::string> &valid_vocab);
+++ const std::vector<absl::string_view> &valid_vocab);
++
++ // Reverts the vocabulary restriction.
++ virtual util::Status ResetVocabulary();
++@@ -230,6 +231,10 @@ class SentencePieceProcessor {
++ virtual util::Status Decode(const std::vector<std::string> &pieces,
++ std::string *detokenized) const;
++
+++ // Given a sequence of pieces, decodes it into a detokenized output.
+++ virtual util::Status Decode(const std::vector<absl::string_view> &pieces,
+++ std::string *detokenized) const;
+++
++ // Given a sequence of ids, decodes it into a detokenized output.
++ virtual util::Status Decode(const std::vector<int> &ids,
++ std::string *detokenized) const;
++@@ -320,16 +325,19 @@ class SentencePieceProcessor {
++ absl::string_view input, int samples, float theta, bool wor,
++ bool include_best, NBestSentencePieceText *samples_spt) const;
++
++-#ifndef SWIG
++ // Calculate entropy of possible tokenisations
++ virtual util::Status CalculateEntropy(absl::string_view input, float theta,
++ float *entropy) const;
++-#endif
++
++ // Given a sequence of pieces, decodes it into SentencePieceText.
+++ // TODO(taku): Remove this API and use std::vector<std::string_view>
++ virtual util::Status Decode(const std::vector<std::string> &pieces,
++ SentencePieceText *spt) const;
++
+++ // Given a sequence of pieces, decodes it into SentencePieceText.
+++ virtual util::Status Decode(const std::vector<absl::string_view> &pieces,
+++ SentencePieceText *spt) const;
+++
++ // Given a sequence of ids, decodes it into SentencePieceText.
++ virtual util::Status Decode(const std::vector<int> &ids,
++ SentencePieceText *spt) const;
++@@ -401,11 +409,17 @@ class SentencePieceProcessor {
++ theta, wor, include_best);
++ }
++
+++ // TODO(taku): Remove this API and use std::vector<std::string_view>
++ virtual std::string DecodePieces(
++ const std::vector<std::string> &pieces) const {
++ DEFINE_SPP_DIRECT_FUNC_IMPL(Decode, std::string, pieces);
++ }
++
+++ virtual std::string DecodePieces(
+++ const std::vector<absl::string_view> &pieces) const {
+++ DEFINE_SPP_DIRECT_FUNC_IMPL(Decode, std::string, pieces);
+++ }
+++
++ virtual std::string DecodeIds(const std::vector<int> &ids) const {
++ DEFINE_SPP_DIRECT_FUNC_IMPL(Decode, std::string, ids);
++ }
++@@ -428,9 +442,13 @@ class SentencePieceProcessor {
++ virtual util::bytes NBestEncodeAsSerializedProto(absl::string_view input,
++ int nbest_size) const;
++
+++ // TODO(taku): Remove this API and use std::vector<std::string_view>
++ virtual util::bytes DecodePiecesAsSerializedProto(
++ const std::vector<std::string> &pieces) const;
++
+++ virtual util::bytes DecodePiecesAsSerializedProto(
+++ const std::vector<absl::string_view> &pieces) const;
+++
++ virtual util::bytes DecodeIdsAsSerializedProto(
++ const std::vector<int> &ids) const;
++
++diff --git a/src/sentencepiece_trainer.h b/src/sentencepiece_trainer.h
++index bb74ab9..b4af6f0 100644
++--- a/src/sentencepiece_trainer.h
+++++ b/src/sentencepiece_trainer.h
++@@ -129,12 +129,12 @@ class SentencePieceTrainer {
++ // with comma-separated values. `field_name` must not be a nested message.
++ // The body of these functions are automatically generated with
++ // data/gen_spec_parser.pl
++- static util::Status SetProtoField(const std::string &name,
++- const std::string &value,
+++ static util::Status SetProtoField(absl::string_view name,
+++ absl::string_view value,
++ TrainerSpec *message);
++
++- static util::Status SetProtoField(const std::string &name,
++- const std::string &value,
+++ static util::Status SetProtoField(absl::string_view name,
+++ absl::string_view value,
++ NormalizerSpec *message);
++
++ // Populates model type from string representation, e.g., "bpe".
++diff --git a/src/spec_parser.h b/src/spec_parser.h
++index b5713fb..de8f72f 100644
++--- a/src/spec_parser.h
+++++ b/src/spec_parser.h
++@@ -25,10 +25,10 @@
++
++ namespace sentencepiece {
++
++-#define PARSE_STRING(param_name) \
++- if (name == #param_name) { \
++- message->set_##param_name(value); \
++- return util::OkStatus(); \
+++#define PARSE_STRING(param_name) \
+++ if (name == #param_name) { \
+++ message->set_##param_name(std::string(value)); \
+++ return util::OkStatus(); \
++ }
++
++ #define PARSE_REPEATED_STRING(param_name) \
++@@ -189,8 +189,8 @@ inline std::string PrintProto(const NormalizerSpec &message,
++ return os.str();
++ }
++
++-util::Status SentencePieceTrainer::SetProtoField(const std::string &name,
++- const std::string &value,
+++util::Status SentencePieceTrainer::SetProtoField(absl::string_view name,
+++ absl::string_view value,
++ TrainerSpec *message) {
++ CHECK_OR_RETURN(message);
++
++@@ -249,8 +249,8 @@ util::Status SentencePieceTrainer::SetProtoField(const std::string &name,
++ << "unknown field name \"" << name << "\" in TrainerSpec.";
++ }
++
++-util::Status SentencePieceTrainer::SetProtoField(const std::string &name,
++- const std::string &value,
+++util::Status SentencePieceTrainer::SetProtoField(absl::string_view name,
+++ absl::string_view value,
++ NormalizerSpec *message) {
++ CHECK_OR_RETURN(message);
++
++diff --git a/src/spm_encode_main.cc b/src/spm_encode_main.cc
++index 4d12a38..b0e508d 100644
++--- a/src/spm_encode_main.cc
+++++ b/src/spm_encode_main.cc
++@@ -92,13 +92,13 @@ int main(int argc, char *argv[]) {
++ absl::flat_hash_map<std::string, int> vocab;
++ sentencepiece::SentencePieceText spt;
++ sentencepiece::NBestSentencePieceText nbest_spt;
++- std::function<void(const std::string &line)> process;
+++ std::function<void(absl::string_view line)> process;
++
++ const int nbest_size = absl::GetFlag(FLAGS_nbest_size);
++ const float alpha = absl::GetFlag(FLAGS_alpha);
++
++ if (absl::GetFlag(FLAGS_generate_vocabulary)) {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.Encode(line, &spt));
++ for (const auto &piece : spt.pieces()) {
++ if (!sp.IsUnknown(piece.id()) && !sp.IsControl(piece.id()))
++@@ -106,47 +106,47 @@ int main(int argc, char *argv[]) {
++ }
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "piece") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.Encode(line, &sps));
++ output->WriteLine(absl::StrJoin(sps, " "));
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "id") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.Encode(line, &ids));
++ output->WriteLine(absl::StrJoin(ids, " "));
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "proto") {
++- process = [&](const std::string &line) { CHECK_OK(sp.Encode(line, &spt)); };
+++ process = [&](absl::string_view line) { CHECK_OK(sp.Encode(line, &spt)); };
++ } else if (absl::GetFlag(FLAGS_output_format) == "sample_piece") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.SampleEncode(line, nbest_size, alpha, &sps));
++ output->WriteLine(absl::StrJoin(sps, " "));
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "sample_id") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.SampleEncode(line, nbest_size, alpha, &ids));
++ output->WriteLine(absl::StrJoin(ids, " "));
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "sample_proto") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.SampleEncode(line, nbest_size, alpha, &spt));
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "nbest_piece") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.NBestEncode(line, nbest_size, &nbest_sps));
++ for (const auto &result : nbest_sps) {
++ output->WriteLine(absl::StrJoin(result, " "));
++ }
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "nbest_id") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.NBestEncode(line, nbest_size, &nbest_ids));
++ for (const auto &result : nbest_ids) {
++ output->WriteLine(absl::StrJoin(result, " "));
++ }
++ };
++ } else if (absl::GetFlag(FLAGS_output_format) == "nbest_proto") {
++- process = [&](const std::string &line) {
+++ process = [&](absl::string_view line) {
++ CHECK_OK(sp.NBestEncode(line, nbest_size, &nbest_spt));
++ };
++ } else {
++diff --git a/src/util.cc b/src/util.cc
++index f99c73a..f54e8ba 100644
++--- a/src/util.cc
+++++ b/src/util.cc
++@@ -244,15 +244,16 @@ std::vector<std::string> StrSplitAsCSV(absl::string_view text) {
++
++ #ifdef OS_WIN
++ namespace win32 {
++-std::wstring Utf8ToWide(const std::string &input) {
++- int output_length =
++- ::MultiByteToWideChar(CP_UTF8, 0, input.c_str(), -1, nullptr, 0);
+++std::wstring Utf8ToWide(absl::string_view input) {
+++ int output_length = ::MultiByteToWideChar(
+++ CP_UTF8, 0, input.data(), static_cast<int>(input.size()), nullptr, 0);
++ output_length = output_length <= 0 ? 0 : output_length - 1;
++ if (output_length == 0) {
++ return L"";
++ }
++ std::unique_ptr<wchar_t[]> input_wide(new wchar_t[output_length + 1]);
++- const int result = ::MultiByteToWideChar(CP_UTF8, 0, input.c_str(), -1,
+++ const int result = ::MultiByteToWideChar(CP_UTF8, 0, input.data(),
+++ static_cast<int>(input.size()),
++ input_wide.get(), output_length + 1);
++ std::wstring output;
++ if (result > 0) {
++@@ -260,24 +261,6 @@ std::wstring Utf8ToWide(const std::string &input) {
++ }
++ return output;
++ }
++-
++-std::string WideToUtf8(const std::wstring &input) {
++- const int output_length = ::WideCharToMultiByte(CP_UTF8, 0, input.c_str(), -1,
++- nullptr, 0, nullptr, nullptr);
++- if (output_length == 0) {
++- return "";
++- }
++-
++- std::unique_ptr<char[]> input_encoded(new char[output_length + 1]);
++- const int result =
++- ::WideCharToMultiByte(CP_UTF8, 0, input.c_str(), -1, input_encoded.get(),
++- output_length + 1, nullptr, nullptr);
++- std::string output;
++- if (result > 0) {
++- output.assign(input_encoded.get());
++- }
++- return output;
++-}
++ } // namespace win32
++ #endif
++ } // namespace sentencepiece
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 15 Jun 2022 02:22:05 +0900
++Subject: Fixed build break.
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ src/common.h | 1 +
++ 1 file changed, 1 insertion(+)
++
++diff --git a/src/common.h b/src/common.h
++index ab07d85..c27c352 100644
++--- a/src/common.h
+++++ b/src/common.h
++@@ -26,6 +26,7 @@
++ #include <vector>
++
++ #include "config.h"
+++#include "third_party/absl/strings/string_view.h"
++
++ #if defined(_WIN32) && !defined(__CYGWIN__)
++ #define OS_WIN
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Mon, 20 Jun 2022 00:55:46 +0900
++Subject: Added ImmutableSentencePiece class
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ src/bpe_model.cc | 6 +-
++ src/model_interface.h | 26 +--
++ src/model_interface_test.cc | 19 +--
++ src/sentencepiece_processor.cc | 173 ++++++++++++-------
++ src/sentencepiece_processor.h | 332 ++++++++++++++++++++++++++----------
++ src/sentencepiece_processor_test.cc | 102 ++++++++---
++ src/unigram_model.cc | 70 ++++----
++ src/unigram_model.h | 15 ++
++ src/unigram_model_test.cc | 114 +++++++------
++ src/util.h | 11 --
++ 10 files changed, 557 insertions(+), 311 deletions(-)
++
++diff --git a/src/bpe_model.cc b/src/bpe_model.cc
++index 22cd115..bc7ada1 100644
++--- a/src/bpe_model.cc
+++++ b/src/bpe_model.cc
++@@ -12,6 +12,8 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
+++#include "bpe_model.h"
+++
++ #include <functional>
++ #include <memory>
++ #include <queue>
++@@ -19,7 +21,6 @@
++ #include <utility>
++ #include <vector>
++
++-#include "bpe_model.h"
++ #include "freelist.h"
++ #include "third_party/absl/container/flat_hash_map.h"
++ #include "util.h"
++@@ -71,8 +72,7 @@ std::vector<std::pair<absl::string_view, int>> Model::SampleEncode(
++ // Reverse merge rules.
++ // key: merged symbol, value: pair of original symbols.
++ absl::flat_hash_map<absl::string_view,
++- std::pair<absl::string_view, absl::string_view>,
++- string_util::string_view_hash>
+++ std::pair<absl::string_view, absl::string_view>>
++ rev_merge;
++
++ // Pre-allocates SymbolPair for efficiency.
++diff --git a/src/model_interface.h b/src/model_interface.h
++index 06b3a65..06e9243 100644
++--- a/src/model_interface.h
+++++ b/src/model_interface.h
++@@ -53,8 +53,8 @@ class ModelProto;
++ // Given a normalized string, returns a sequence of sentence pieces with ids.
++ class ModelInterface {
++ public:
++- using PieceToIdMap = absl::flat_hash_map<absl::string_view, int,
++- string_util::string_view_hash>;
+++ using PieceToIdMap = absl::flat_hash_map<absl::string_view, int>;
+++ // string_util::string_view_hash>;
++
++ absl::string_view unk_piece() const;
++ absl::string_view bos_piece() const;
++@@ -77,19 +77,6 @@ class ModelInterface {
++ return matcher_.get();
++ }
++
++- // Sets the encoder version. Currently only unigram has an optimized encoder.
++- // The optimized version is always used by default if there is one, so
++- // normally users do not need to call this function. This function is provided
++- // just in case that a user want to manually choose which encoder version to
++- // use.
++- virtual util::Status SetEncoderVersion(EncoderVersion encoder_version) {
++- encoder_version_ = encoder_version;
++- return util::OkStatus();
++- }
++-
++- // Returns the current encoder version in use.
++- virtual EncoderVersion GetEncoderVersion() const { return encoder_version_; }
++-
++ // Given a normalized string, returns a sequence of sentence pieces with ids.
++ // The concatenation of pieces must be the same as `normalized`.
++ virtual EncodeResult Encode(absl::string_view normalized) const = 0;
++@@ -123,10 +110,9 @@ class ModelInterface {
++ }
++
++ // Calculates the entropy of the segmentation lattice with inverse temperature
++- // `theta`.
++- // Uses a novel dynamic program to calculate the entropy.
+++ // `alpha`. Uses a novel dynamic program to calculate the entropy.
++ virtual float CalculateEntropy(absl::string_view normalized,
++- float theta) const {
+++ float alpha) const {
++ LOG(ERROR) << "Not implemented.";
++ return 0.0;
++ }
++@@ -256,10 +242,6 @@ class ModelInterface {
++ // unknown id.
++ int unk_id_ = 0;
++
++- // The encoder version. Currently it is only effective for unigram model but
++- // ignored by other models.
++- EncoderVersion encoder_version_ = EncoderVersion::kOptimized;
++-
++ // status.
++ util::Status status_;
++ };
++diff --git a/src/model_interface_test.cc b/src/model_interface_test.cc
++index 69ee4e6..09e41d3 100644
++--- a/src/model_interface_test.cc
+++++ b/src/model_interface_test.cc
++@@ -12,8 +12,9 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
++-#include "model_factory.h"
++ #include "model_interface.h"
+++
+++#include "model_factory.h"
++ #include "testharness.h"
++ #include "third_party/absl/container/flat_hash_map.h"
++ #include "util.h"
++@@ -481,22 +482,6 @@ TEST(ModelInterfaceTest, PieceToByteTest) {
++ EXPECT_EQ(PieceToByte("a"), -1);
++ }
++
++-TEST(ModelInterfaceTest, SetEncoderVersion) {
++- for (const auto type : kModelTypes) {
++- ModelProto model_proto = MakeBaseModelProto(type);
++- AddPiece(&model_proto, "a");
++- AddPiece(&model_proto, "b");
++- auto model = ModelFactory::Create(model_proto);
++-
++- // Verify the default encoder version.
++- EXPECT_EQ(EncoderVersion::kOptimized, model->GetEncoderVersion());
++-
++- // Set the encoder version to original and verify.
++- EXPECT_TRUE(model->SetEncoderVersion(EncoderVersion::kOriginal).ok());
++- EXPECT_EQ(EncoderVersion::kOriginal, model->GetEncoderVersion());
++- }
++-}
++-
++ TEST(ModelInterfaceTest, VerifyOutputsEquivalent) {
++ for (const auto type : kModelTypes) {
++ ModelProto model_proto = MakeBaseModelProto(type);
++diff --git a/src/sentencepiece_processor.cc b/src/sentencepiece_processor.cc
++index 331fc90..a6f5395 100644
++--- a/src/sentencepiece_processor.cc
+++++ b/src/sentencepiece_processor.cc
++@@ -56,6 +56,112 @@ std::vector<absl::string_view> ToPieceArray(const std::vector<std::string> &v) {
++ }
++ } // namespace
++
+++ImmutableSentencePieceText::ImmutableSentencePieceText() {}
+++ImmutableSentencePieceText::~ImmutableSentencePieceText() {}
+++
+++ImmutableSentencePieceText::ImmutableSentencePieceText(
+++ const SentencePieceText &spt)
+++ : spt_(&spt) {}
+++
+++ImmutableSentencePieceText::ImmutableSentencePiece::ImmutableSentencePiece(
+++ const SentencePieceText_SentencePiece &sp)
+++ : sp_(&sp) {}
+++
+++absl::string_view ImmutableSentencePieceText::ImmutableSentencePiece::piece()
+++ const {
+++ return sp_->piece();
+++}
+++
+++absl::string_view ImmutableSentencePieceText::ImmutableSentencePiece::surface()
+++ const {
+++ return sp_->surface();
+++}
+++
+++uint32_t ImmutableSentencePieceText::ImmutableSentencePiece::id() const {
+++ return sp_->id();
+++}
+++
+++uint32_t ImmutableSentencePieceText::ImmutableSentencePiece::begin() const {
+++ return sp_->begin();
+++}
+++
+++uint32_t ImmutableSentencePieceText::ImmutableSentencePiece::end() const {
+++ return sp_->end();
+++}
+++
+++std::vector<ImmutableSentencePieceText::ImmutableSentencePiece>
+++ImmutableSentencePieceText::pieces() const {
+++ std::vector<ImmutableSentencePieceText::ImmutableSentencePiece> pieces;
+++ if (spt_ == nullptr) return pieces;
+++ pieces.reserve(spt_->pieces_size());
+++ for (int i = 0; i < spt_->pieces_size(); ++i)
+++ pieces[i] = ImmutableSentencePiece(spt_->pieces(i));
+++ return pieces;
+++}
+++
+++size_t ImmutableSentencePieceText::pieces_size() const {
+++ return spt_ ? spt_->pieces_size() : 0;
+++}
+++
+++ImmutableSentencePieceText::ImmutableSentencePiece
+++ImmutableSentencePieceText::pieces(int index) const {
+++ return ImmutableSentencePieceText::ImmutableSentencePiece(
+++ spt_->pieces(index));
+++}
+++
+++absl::string_view ImmutableSentencePieceText::text() const {
+++ return spt_ ? spt_->text() : "";
+++}
+++
+++float ImmutableSentencePieceText::score() const {
+++ return spt_ ? spt_->score() : 0.0;
+++}
+++
+++SentencePieceText *ImmutableSentencePieceText::mutable_proto() {
+++ if (rep_ == nullptr) {
+++ rep_ = std::make_shared<SentencePieceText>();
+++ spt_ = rep_.get();
+++ }
+++ return rep_.get();
+++}
+++
+++std::string ImmutableSentencePieceText::SerializeAsString() const {
+++ return spt_ ? spt_->SerializeAsString() : "";
+++}
+++
+++ImmutableNBestSentencePieceText::ImmutableNBestSentencePieceText() {}
+++ImmutableNBestSentencePieceText::~ImmutableNBestSentencePieceText() {}
+++
+++size_t ImmutableNBestSentencePieceText::nbests_size() const {
+++ return rep_ ? rep_->nbests_size() : 0;
+++}
+++
+++ImmutableSentencePieceText ImmutableNBestSentencePieceText::nbests(
+++ int index) const {
+++ return ImmutableSentencePieceText(rep_->nbests(index));
+++}
+++
+++std::vector<ImmutableSentencePieceText>
+++ImmutableNBestSentencePieceText::nbests() const {
+++ std::vector<ImmutableSentencePieceText> nbests;
+++ if (rep_ == nullptr) return nbests;
+++ nbests.reserve(rep_->nbests_size());
+++ for (int i = 0; i < rep_->nbests_size(); ++i)
+++ nbests[i] = ImmutableSentencePieceText(rep_->nbests(i));
+++ return nbests;
+++}
+++
+++NBestSentencePieceText *ImmutableNBestSentencePieceText::mutable_proto() {
+++ if (rep_ == nullptr) {
+++ rep_ = std::make_shared<NBestSentencePieceText>();
+++ }
+++ return rep_.get();
+++}
+++
+++std::string ImmutableNBestSentencePieceText::SerializeAsString() const {
+++ return rep_ ? rep_->SerializeAsString() : "";
+++}
+++
++ SentencePieceProcessor::SentencePieceProcessor() {}
++ SentencePieceProcessor::~SentencePieceProcessor() {}
++
++@@ -124,15 +230,6 @@ util::Status SentencePieceProcessor::Load(
++ return util::OkStatus();
++ }
++
++-util::Status SentencePieceProcessor::SetEncoderVersion(
++- EncoderVersion encoder_version) {
++- return model_->SetEncoderVersion(encoder_version);
++-}
++-
++-EncoderVersion SentencePieceProcessor::GetEncoderVersion() const {
++- return model_->GetEncoderVersion();
++-}
++-
++ util::Status SentencePieceProcessor::SetEncodeExtraOptions(
++ absl::string_view extra_options) {
++ return ParseExtraOptions(extra_options, &encode_extra_options_);
++@@ -348,14 +445,14 @@ util::Status SentencePieceProcessor::SampleEncode(absl::string_view input,
++ }
++
++ util::Status SentencePieceProcessor::SampleEncodeAndScore(
++- absl::string_view input, int num_samples, float theta, bool wor,
+++ absl::string_view input, int num_samples, float alpha, bool wor,
++ bool include_best,
++ std::vector<std::pair<std::vector<std::string>, float>> *pieces) const {
++ CHECK_OR_RETURN_STATUS_STL(pieces);
++
++ NBestSentencePieceText spt;
++ RETURN_IF_ERROR(
++- SampleEncodeAndScore(input, num_samples, theta, wor, include_best, &spt));
+++ SampleEncodeAndScore(input, num_samples, alpha, wor, include_best, &spt));
++
++ pieces->clear();
++ pieces->reserve(spt.nbests_size());
++@@ -373,14 +470,14 @@ util::Status SentencePieceProcessor::SampleEncodeAndScore(
++ }
++
++ util::Status SentencePieceProcessor::SampleEncodeAndScore(
++- absl::string_view input, int num_samples, float theta, bool wor,
+++ absl::string_view input, int num_samples, float alpha, bool wor,
++ bool include_best,
++ std::vector<std::pair<std::vector<int>, float>> *ids) const {
++ CHECK_OR_RETURN_STATUS_STL(ids);
++
++ NBestSentencePieceText spt;
++ RETURN_IF_ERROR(
++- SampleEncodeAndScore(input, num_samples, theta, wor, include_best, &spt));
+++ SampleEncodeAndScore(input, num_samples, alpha, wor, include_best, &spt));
++
++ ids->clear();
++ ids->reserve(spt.nbests_size());
++@@ -568,7 +665,7 @@ util::Status SentencePieceProcessor::SampleEncode(
++ }
++
++ util::Status SentencePieceProcessor::SampleEncodeAndScore(
++- absl::string_view input, int samples, float theta, bool wor,
+++ absl::string_view input, int samples, float alpha, bool wor,
++ bool include_best, NBestSentencePieceText *samples_spt) const {
++ CHECK_OR_RETURN(model_->IsSampleEncodeAndScoreAvailable())
++ << "SampleEncodeAndScore is not available for the current model.";
++@@ -576,7 +673,7 @@ util::Status SentencePieceProcessor::SampleEncodeAndScore(
++ std::vector<size_t> norm_to_orig;
++ RETURN_IF_ERROR(normalizer_->Normalize(input, &normalized, &norm_to_orig));
++
++- const auto results = model_->SampleEncodeAndScore(normalized, theta, samples,
+++ const auto results = model_->SampleEncodeAndScore(normalized, alpha, samples,
++ wor, include_best);
++ CHECK_OR_RETURN(!results.empty())
++ << "SampleEncodeAndScore returns empty result.";
++@@ -592,7 +689,7 @@ util::Status SentencePieceProcessor::SampleEncodeAndScore(
++ }
++
++ util::Status SentencePieceProcessor::CalculateEntropy(absl::string_view input,
++- float theta,
+++ float alpha,
++ float *entropy) const {
++ CHECK_OR_RETURN(model_->IsCalculateEntropyAvailable())
++ << "CalculateEntropy is not available for the current model.";
++@@ -600,7 +697,7 @@ util::Status SentencePieceProcessor::CalculateEntropy(absl::string_view input,
++ std::vector<size_t> norm_to_orig;
++ RETURN_IF_ERROR(normalizer_->Normalize(input, &normalized, &norm_to_orig));
++
++- *entropy = model_->CalculateEntropy(normalized, theta);
+++ *entropy = model_->CalculateEntropy(normalized, alpha);
++ return util::OkStatus();
++ }
++
++@@ -770,48 +867,6 @@ util::Status SentencePieceProcessor::Decode(const std::vector<int> &ids,
++ return Decode(pieces, spt);
++ }
++
++-std::string SentencePieceProcessor::EncodeAsSerializedProto(
++- absl::string_view input) const {
++- SentencePieceText spt;
++- if (!Encode(input, &spt).ok()) return "";
++- return spt.SerializeAsString();
++-}
++-
++-std::string SentencePieceProcessor::SampleEncodeAsSerializedProto(
++- absl::string_view input, int nbest_size, float alpha) const {
++- SentencePieceText spt;
++- if (!SampleEncode(input, nbest_size, alpha, &spt).ok()) return "";
++- return spt.SerializeAsString();
++-}
++-
++-std::string SentencePieceProcessor::NBestEncodeAsSerializedProto(
++- absl::string_view input, int nbest_size) const {
++- NBestSentencePieceText spt;
++- if (!NBestEncode(input, nbest_size, &spt).ok()) return "";
++- return spt.SerializeAsString();
++-}
++-
++-std::string SentencePieceProcessor::DecodePiecesAsSerializedProto(
++- const std::vector<std::string> &pieces) const {
++- SentencePieceText spt;
++- if (!Decode(pieces, &spt).ok()) return "";
++- return spt.SerializeAsString();
++-}
++-
++-std::string SentencePieceProcessor::DecodePiecesAsSerializedProto(
++- const std::vector<absl::string_view> &pieces) const {
++- SentencePieceText spt;
++- if (!Decode(pieces, &spt).ok()) return "";
++- return spt.SerializeAsString();
++-}
++-
++-std::string SentencePieceProcessor::DecodeIdsAsSerializedProto(
++- const std::vector<int> &ids) const {
++- SentencePieceText spt;
++- if (!Decode(ids, &spt).ok()) return "";
++- return spt.SerializeAsString();
++-}
++-
++ #define CHECK_STATUS_OR_RETURN_DEFAULT(value) \
++ if (!status().ok()) { \
++ LOG(ERROR) << status().message() << "\nReturns default value " << value; \
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index 8c72656..51c5b3b 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -29,11 +29,6 @@ using std::string_view;
++ #endif // SWIG
++
++ namespace sentencepiece {
++-
++-#ifndef SWIG
++-using EncodeResult = std::vector<std::pair<absl::string_view, int>>;
++-#endif // SWIG
++-
++ namespace util {
++
++ enum class StatusCode : int {
++@@ -107,17 +102,17 @@ class Status {
++ // sp.Load("//path/to/model");
++ //
++ // vector<string> sps;
++-// sp.Encode("hello world.", &sps);
+++// sp.Encode("hello world.", &sps).IgnoreError();
++ //
++ // vector<int> ids;
++-// sp.Encode("hello world.", &ids);
+++// sp.Encode("hello world.", &ids).IgnoreError();
++ //
++ // string detok;
++ // sp.Decode(sps, &detok);
++-// CHECK_EQ("hello world.", detok);
+++// CHECK_EQ("hello world.", detok).IgnoreError();
++ //
++ // sp.Decode(ids, &detok);
++-// CHECK_EQ("hello world.", detok);
+++// CHECK_EQ("hello world.", detok).IgnoreError();
++ //
++ // We can also use SentencePieceText which manages the byte-offsets
++ // between user input (output) and internal sentence pieces.
++@@ -144,16 +139,6 @@ namespace normalizer {
++ class Normalizer;
++ } // namespace normalizer
++
++-#ifndef SWIG
++-// Defines the multiple versions of encoder within each model. Currently only
++-// the Unigram model has an optimized encoder.
++-enum class EncoderVersion {
++- kOptimized, // The optimized encoder (default).
++- kOriginal // The original encoder (user may choose to fall back to this
++- // just in case).
++-};
++-#endif
++-
++ #ifndef SWIGGO
++ namespace util {
++ // Redefine std::string for serialized_proto interface as Python's string is
++@@ -161,7 +146,87 @@ namespace util {
++ // with SWIG's typemap.
++ using bytes = std::string;
++ } // namespace util
++-#endif
+++#endif // SWIGGO
+++
+++class NBestSentencePieceText;
+++class ModelInterface;
+++class SentencePieceText;
+++class SentencePieceText_SentencePiece;
+++
+++// Wrapper class of SentencePieceText
+++// This wrapper only allows an immutable access to the proto and
+++// hides the actual implementation of protobuf.
+++// See sentencepiece.proto for the details of this class.
+++class ImmutableSentencePieceText {
+++ public:
+++ ImmutableSentencePieceText();
+++ virtual ~ImmutableSentencePieceText();
+++
+++ class ImmutableSentencePiece {
+++ public:
+++ ~ImmutableSentencePiece() = default;
+++ absl::string_view piece() const;
+++ absl::string_view surface() const;
+++ uint32_t id() const;
+++ uint32_t begin() const;
+++ uint32_t end() const;
+++
+++ friend class ImmutableSentencePieceText;
+++
+++ private:
+++ ImmutableSentencePiece() = default;
+++ explicit ImmutableSentencePiece(const SentencePieceText_SentencePiece &sp);
+++ const SentencePieceText_SentencePiece *sp_ = nullptr;
+++ };
+++
+++ std::vector<ImmutableSentencePiece> pieces() const;
+++ size_t pieces_size() const;
+++ ImmutableSentencePiece pieces(int index) const;
+++ absl::string_view text() const;
+++ float score() const;
+++
+++ std::string SerializeAsString() const;
+++
+++ // Returns the actual mutable proto.
+++ // Do not use this outside of SentencePieceProcessor, as
+++ // it returns the raw pointer managed by the shared_ptr.
+++ SentencePieceText *mutable_proto();
+++
+++ friend class ImmutableNBestSentencePieceText;
+++ friend class SentencePieceProcessor;
+++
+++ private:
+++ explicit ImmutableSentencePieceText(const SentencePieceText &spt);
+++ const SentencePieceText *spt_ = nullptr;
+++ std::shared_ptr<SentencePieceText> rep_;
+++};
+++
+++// Wrapper class of SentencePieceText
+++// This wrapper only allows an immutable access to the proto and
+++// hides the actual implementation of protobuf.
+++// See sentencepiece.proto for the details of this class.
+++class ImmutableNBestSentencePieceText {
+++ public:
+++ ImmutableNBestSentencePieceText();
+++ virtual ~ImmutableNBestSentencePieceText();
+++
+++ std::vector<ImmutableSentencePieceText> nbests() const;
+++
+++ size_t nbests_size() const;
+++ ImmutableSentencePieceText nbests(int index) const;
+++
+++ std::string SerializeAsString() const;
+++
+++ // Returns the actual mutable proto.
+++ // Do not use this outside of SentencePieceProcessor, as
+++ // it returns the raw pointer managed by the shared_ptr.
+++ NBestSentencePieceText *mutable_proto();
+++
+++ friend class SentencePieceProcessor;
+++
+++ private:
+++ std::shared_ptr<NBestSentencePieceText> rep_;
+++};
++
++ class SentencePieceProcessor {
++ public:
++@@ -217,7 +282,7 @@ class SentencePieceProcessor {
++ int threshold);
++
++ //////////////////////////////////////////////////////////////
++- // Simple API.
+++ // Simple Encode and Decode API.
++ //
++ // Given a UTF8 input, encodes it into a sequence of sentence pieces.
++ virtual util::Status Encode(absl::string_view input,
++@@ -239,18 +304,9 @@ class SentencePieceProcessor {
++ virtual util::Status Decode(const std::vector<int> &ids,
++ std::string *detokenized) const;
++
++-#ifndef SWIG
++- // Sets the encoder version. Normally users do not need to call this function.
++- // But they can call this fucntion just in case if they want to fall back to
++- // the original encoder.
++- virtual util::Status SetEncoderVersion(EncoderVersion encoder_version);
++-
++- // Returns the current encoder version in use.
++- virtual EncoderVersion GetEncoderVersion() const;
++-#endif
++-
++ //////////////////////////////////////////////////////////////
++ // NBest API.
+++ //
++ // Same as Encode, but returns nbest results.
++ virtual util::Status NBestEncode(
++ absl::string_view input, int nbest_size,
++@@ -262,24 +318,24 @@ class SentencePieceProcessor {
++
++ //////////////////////////////////////////////////////////////
++ // Sampling API.
+++ //
++ // Unigram and BPE support sampling mode.
++ // - Unigram (--model_type=unigram):
++- // When `nbest_size` is positive value, approximately samples one
++- // segmentation from nbest candidates. When `nbest_size` is negative value,
++- // samples one segmentation from the hypotheses (Lattice) according to the
++- // generation probabilities using forward-filtering and backward-sampling
++- // algorithm. `alpha` is a smoothing parameter. The best segmentation
++- // (Viterbi segmentation) is more likely sampled when setting larger
++- // alpha. When alpha is 0.0, one segmentation is uniformly sampled from the
++- // nbest or lattice.
++- // `nbest_size` and `alpha` correspond to parameters `l` and `alpha`
+++ // `nbest_size`: When `nbest_size` is positive value, approximately samples
+++ // one segmentation from nbest candidates. When `nbest_size` is negative
+++ // value, samples one segmentation from the hypotheses (Lattice) according to
+++ // the generation probabilities using forward-filtering and backward-sampling
+++ // algorithm.
+++ // `alpha`: Smoothing parameter (inverse temperature). The best segmentation
+++ // (Viterbi segmentation) is more likely sampled when setting larger alpha.
+++ // When alpha is 0.0, one segmentation is uniformly sampled from the nbest or
+++ // lattice. `nbest_size` and `alpha` correspond to parameters `l` and `alpha`
++ // in https://arxiv.org/abs/1804.10959 (nbest_size < 0 means l = infinity)
++ //
++ // - BPE (--model_type=bpe):
++- // `alpha` is the dropout probability `p` of bpe merge operations
++- // in https://arxiv.org/abs/1910.13267
++- // Nbest-based sampling is not supported so nbest_size parameter is ignored in
++- // BPE.
+++ // `alpha`: The dropout probability `p` of bpe merge operations in
+++ // https://arxiv.org/abs/1910.13267 Nbest-based sampling is not supported so
+++ // nbest_size parameter is ignored in BPE.
++ virtual util::Status SampleEncode(absl::string_view input, int nbest_size,
++ float alpha,
++ std::vector<std::string> *pieces) const;
++@@ -290,74 +346,104 @@ class SentencePieceProcessor {
++
++ //////////////////////////////////////////////////////////////
++ // SampleEncodeAndScore API.
++- // Similar to SampleEncode, but returns samples results.
+++ //
+++ // Sample `samples` many tokenisations from the segmentation lattice.
+++ // These methods are only available in model_type=unigram.
+++ //
+++ // `alpha`: smoothing parameter (inverse temperature). The same as `alpha` in
+++ // `Sample` method.
+++ // 'wor`: If `wor` is true, the samples are taken without replacement, and the
+++ // scores are the inclusion probabilities of the elements in the sample;
+++ // otherwise the samples are taken with replacement and the scores are the
+++ // log-probs of sample elements
+++ // `include_best`: If `include_best` is true, the best tokenisation is always
+++ // included in the sample, and the remaining elements are sampled excluding
+++ // the best.
++ virtual util::Status SampleEncodeAndScore(
++- absl::string_view input, int num_samples, float theta, bool wor,
+++ absl::string_view input, int num_samples, float alpha, bool wor,
++ bool include_best,
++ std::vector<std::pair<std::vector<std::string>, float>> *pieces) const;
++
++ // Same as above, but returns a sequence of ids.
++ virtual util::Status SampleEncodeAndScore(
++- absl::string_view input, int num_samples, float theta, bool wor,
+++ absl::string_view input, int num_samples, float alpha, bool wor,
++ bool include_best,
++ std::vector<std::pair<std::vector<int>, float>> *ids) const;
++
+++ //////////////////////////////////////////////////////////////
+++ // Entropy API.
+++ //
+++ // This only available in model_type=unigram.
+++ // Calculate entropy of possible tokenisations
+++ virtual util::Status CalculateEntropy(absl::string_view input, float alpha,
+++ float *entropy) const;
+++
++ //////////////////////////////////////////////////////////////
++ // Advanced API returning SentencePieceText, which manages
++ // utf8-byte alignments between user-input/detokenized text
++ // and internal sentencepiece sequence.
++ //
++ // Given a UTF8 input, encodes it into SentencePieceText.
+++ //
+++ // When using these APIs, sentencepiece.pb.h header files must be included.
+++ // We can also use ImutableSentencePieceText as follows.
+++ //
+++ // ImmutableSentencePieceText spt;
+++ // Encode("hello", spt.mutable_proto()).IgnoreError();
+++ // std::cout << spt.pieces_size() << std::endl;
++ virtual util::Status Encode(absl::string_view input,
++ SentencePieceText *spt) const;
++
++- // Same as above, but returns NBestSentencePieceText.
++ virtual util::Status NBestEncode(absl::string_view input, int nbest_size,
++ NBestSentencePieceText *nbest_spt) const;
++
++- // Same as above, but samples one segmentation from the hypotheses
++- // (Lattice).
++ virtual util::Status SampleEncode(absl::string_view input, int nbest_size,
++ float alpha, SentencePieceText *spt) const;
++
++- // Samples N segmentation and returns the scores as well
++ virtual util::Status SampleEncodeAndScore(
++- absl::string_view input, int samples, float theta, bool wor,
+++ absl::string_view input, int samples, float alpha, bool wor,
++ bool include_best, NBestSentencePieceText *samples_spt) const;
++
++- // Calculate entropy of possible tokenisations
++- virtual util::Status CalculateEntropy(absl::string_view input, float theta,
++- float *entropy) const;
++-
++- // Given a sequence of pieces, decodes it into SentencePieceText.
++- // TODO(taku): Remove this API and use std::vector<std::string_view>
+++ // DEPRECATED: Remove this API and use std::vector<std::string_view>
++ virtual util::Status Decode(const std::vector<std::string> &pieces,
++ SentencePieceText *spt) const;
++
++- // Given a sequence of pieces, decodes it into SentencePieceText.
++ virtual util::Status Decode(const std::vector<absl::string_view> &pieces,
++ SentencePieceText *spt) const;
++
++- // Given a sequence of ids, decodes it into SentencePieceText.
++ virtual util::Status Decode(const std::vector<int> &ids,
++ SentencePieceText *spt) const;
++
++- //////////////////////////////////////////////////////////////
++- // Handy methods that return the result directly.
++- // These functions ignore internal errors.
++ #ifdef SWIG
++-#define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
++- OutType output; \
++- const auto _status = FuncName(__VA_ARGS__, &output); \
++- if (!_status.ok()) throw _status; \
++- return output;
+++#define SPP_SWIG_CHECK_AND_THROW \
+++ if (!status.ok()) throw status;
++ #else
+++#define SPP_SWIG_CHECK_AND_THROW \
+++ if (!status.ok()) { \
+++ }
+++#endif // SWIG
+++
++ #define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++- FuncName(__VA_ARGS__, &output).IgnoreError(); \
+++ const auto status = FuncName(__VA_ARGS__, &output); \
+++ SPP_SWIG_CHECK_AND_THROW; \
+++ return output;
+++
+++#define DEFINE_SPP_SERIALIZED_PROTO_IMPL(FuncName, OutType, ...) \
+++ OutType output; \
+++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
+++ SPP_SWIG_CHECK_AND_THROW; \
+++ return output.SerializeAsString();
+++
+++#define DEFINE_SPP_IMMUTABLE_PROTO_IMPL(FuncName, OutType, ...) \
+++ OutType output; \
+++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
+++ SPP_SWIG_CHECK_AND_THROW; \
++ return output;
++-#endif
++
+++ //////////////////////////////////////////////////////////////
+++ // Handy methods that return the result directly.
+++ // These functions ignore internal errors.
++ virtual std::vector<std::string> EncodeAsPieces(
++ absl::string_view input) const {
++ DEFINE_SPP_DIRECT_FUNC_IMPL(Encode, std::vector<std::string>, input);
++@@ -395,21 +481,21 @@ class SentencePieceProcessor {
++
++ virtual std::vector<std::pair<std::vector<std::string>, float>>
++ SampleEncodeAndScoreAsPieces(absl::string_view input, int num_samples,
++- float theta, bool wor, bool include_best) const {
+++ float alpha, bool wor, bool include_best) const {
++ using _T = std::vector<std::pair<std::vector<std::string>, float>>;
++ DEFINE_SPP_DIRECT_FUNC_IMPL(SampleEncodeAndScore, _T, input, num_samples,
++- theta, wor, include_best);
+++ alpha, wor, include_best);
++ }
++
++ virtual std::vector<std::pair<std::vector<int>, float>>
++ SampleEncodeAndScoreAsIds(absl::string_view input, int num_samples,
++- float theta, bool wor, bool include_best) const {
+++ float alpha, bool wor, bool include_best) const {
++ using _T = std::vector<std::pair<std::vector<int>, float>>;
++ DEFINE_SPP_DIRECT_FUNC_IMPL(SampleEncodeAndScore, _T, input, num_samples,
++- theta, wor, include_best);
+++ alpha, wor, include_best);
++ }
++
++- // TODO(taku): Remove this API and use std::vector<std::string_view>
+++ // DEPRECATED: Remove this API and use std::vector<std::string_view>
++ virtual std::string DecodePieces(
++ const std::vector<std::string> &pieces) const {
++ DEFINE_SPP_DIRECT_FUNC_IMPL(Decode, std::string, pieces);
++@@ -424,33 +510,104 @@ class SentencePieceProcessor {
++ DEFINE_SPP_DIRECT_FUNC_IMPL(Decode, std::string, ids);
++ }
++
++- virtual float CalculateEntropy(absl::string_view text, float theta) const {
++- DEFINE_SPP_DIRECT_FUNC_IMPL(CalculateEntropy, float, text, theta);
+++ virtual float CalculateEntropy(absl::string_view text, float alpha) const {
+++ DEFINE_SPP_DIRECT_FUNC_IMPL(CalculateEntropy, float, text, alpha);
++ }
++
++-#undef DEFINE_SPP_DIRECT_FUNC_IMPL
++-
+++ //////////////////////////////////////////////////////////////
+++ // SerializedProto API. (DEPRECATED). Use ImmutableProto API.
++ // They are used in Python interface. Returns serialized proto.
++ // In python module, we can get access to the full Proto after
++ // deserialzing the returned byte sequence.
++- virtual util::bytes EncodeAsSerializedProto(absl::string_view input) const;
+++ virtual util::bytes EncodeAsSerializedProto(absl::string_view input) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(Encode, ImmutableSentencePieceText, input);
+++ }
++
++ virtual util::bytes SampleEncodeAsSerializedProto(absl::string_view input,
++ int nbest_size,
++- float alpha) const;
+++ float alpha) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(SampleEncode, ImmutableSentencePieceText,
+++ input, nbest_size, alpha);
+++ }
++
++ virtual util::bytes NBestEncodeAsSerializedProto(absl::string_view input,
++- int nbest_size) const;
+++ int nbest_size) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(
+++ NBestEncode, ImmutableNBestSentencePieceText, input, nbest_size);
+++ }
+++
+++ virtual util::bytes SampleEncodeAndScoreAsSerializedProto(
+++ absl::string_view input, int samples, float alpha, bool wor,
+++ bool include_best, int nbest_size) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(SampleEncodeAndScore,
+++ ImmutableNBestSentencePieceText, input,
+++ samples, alpha, wor, include_best);
+++ }
++
++ // TODO(taku): Remove this API and use std::vector<std::string_view>
++ virtual util::bytes DecodePiecesAsSerializedProto(
++- const std::vector<std::string> &pieces) const;
+++ const std::vector<std::string> &pieces) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(Decode, ImmutableSentencePieceText,
+++ pieces);
+++ }
++
++ virtual util::bytes DecodePiecesAsSerializedProto(
++- const std::vector<absl::string_view> &pieces) const;
+++ const std::vector<absl::string_view> &pieces) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(Decode, ImmutableSentencePieceText,
+++ pieces);
+++ }
++
++ virtual util::bytes DecodeIdsAsSerializedProto(
++- const std::vector<int> &ids) const;
+++ const std::vector<int> &ids) const {
+++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(Decode, ImmutableSentencePieceText, ids);
+++ }
+++
+++ //////////////////////////////////////////////////////////////
+++ // ImmutableProto API.
+++ virtual ImmutableSentencePieceText EncodeAsImmutableProto(
+++ absl::string_view input) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(Encode, ImmutableSentencePieceText, input);
+++ }
+++
+++ virtual ImmutableSentencePieceText SampleEncodeAsImmutableProto(
+++ absl::string_view input, int nbest_size, float alpha) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(SampleEncode, ImmutableSentencePieceText,
+++ input, nbest_size, alpha);
+++ }
+++
+++ virtual ImmutableNBestSentencePieceText NBestEncodeAsImmutableProto(
+++ absl::string_view input, int nbest_size) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(
+++ NBestEncode, ImmutableNBestSentencePieceText, input, nbest_size);
+++ }
+++
+++ virtual ImmutableNBestSentencePieceText SampleEncodeAndScoreAsImmutableProto(
+++ absl::string_view input, int samples, float alpha, bool wor,
+++ bool include_best, int nbest_size) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(SampleEncodeAndScore,
+++ ImmutableNBestSentencePieceText, input,
+++ samples, alpha, wor, include_best);
+++ }
+++
+++ // TODO(taku): Remove this API and use std::vector<std::string_view>
+++ virtual ImmutableSentencePieceText DecodePiecesAsImmutableProto(
+++ const std::vector<std::string> &pieces) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(Decode, ImmutableSentencePieceText, pieces);
+++ }
+++
+++ virtual ImmutableSentencePieceText DecodePiecesAsImmutableProto(
+++ const std::vector<absl::string_view> &pieces) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(Decode, ImmutableSentencePieceText, pieces);
+++ }
+++
+++ virtual ImmutableSentencePieceText DecodeIdsAsImmutableProto(
+++ const std::vector<int> &ids) const {
+++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(Decode, ImmutableSentencePieceText, ids);
+++ }
+++
+++#undef DEFINE_SPP_DIRECT_FUNC_IMPL
+++#undef DEFINE_SPP_SERIALIZED_PROTO_IMPL
+++#undef DEFINE_SPP_IMMUTABLE_PROTO_IMPL
++
++ //////////////////////////////////////////////////////////////
++ // Vocabulary management methods.
++@@ -467,7 +624,8 @@ class SentencePieceProcessor {
++ virtual const std::string &IdToPiece(int id) const;
++
++ // Returns the score of `id`.
++- // Usually score is an emission log probability of unigram language model.
+++ // Usually score is an emission log probability of unigram language
+++ // model.
++ virtual float GetScore(int id) const;
++
++ // Returns true if `id` is unknown symbol.
++@@ -506,7 +664,7 @@ class SentencePieceProcessor {
++
++ // Allows injection of a normalizer instance. `normalizer` is moved.
++ void SetNormalizer(std::unique_ptr<normalizer::Normalizer> &&normalizer);
++-#endif
+++#endif // SWIG
++
++ // Returns immutable model proto. Useful to obtain extended
++ // or experimental parameters encoded in model_proto.
++diff --git a/src/sentencepiece_processor_test.cc b/src/sentencepiece_processor_test.cc
++index d57ab5a..ed651f7 100644
++--- a/src/sentencepiece_processor_test.cc
+++++ b/src/sentencepiece_processor_test.cc
++@@ -12,6 +12,8 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
+++#include "sentencepiece_processor.h"
+++
++ #include <utility>
++
++ #include "builder.h"
++@@ -20,7 +22,6 @@
++ #include "normalizer.h"
++ #include "sentencepiece.pb.h"
++ #include "sentencepiece_model.pb.h"
++-#include "sentencepiece_processor.h"
++ #include "sentencepiece_trainer.h"
++ #include "testharness.h"
++ #include "third_party/absl/container/flat_hash_map.h"
++@@ -551,10 +552,9 @@ TEST(SentencepieceProcessorTest, DecodeTest) {
++ int GetPieceSize() const override { return 7; }
++
++ int PieceToId(absl::string_view piece) const override {
++- static absl::flat_hash_map<absl::string_view, int,
++- string_util::string_view_hash>
++- kMap = {{"<unk>", 0}, {"<s>", 1}, {"</s>", 2}, {WS "ABC", 3},
++- {WS "DE", 4}, {"F", 5}, {"G" WS "H", 6}};
+++ static absl::flat_hash_map<absl::string_view, int> kMap = {
+++ {"<unk>", 0}, {"<s>", 1}, {"</s>", 2}, {WS "ABC", 3},
+++ {WS "DE", 4}, {"F", 5}, {"G" WS "H", 6}};
++ return port::FindWithDefault(kMap, piece, 0);
++ }
++
++@@ -719,10 +719,9 @@ TEST(SentencepieceProcessorTest, DummyPrefixDecodeTest) {
++ int GetPieceSize() const override { return 7; }
++
++ int PieceToId(absl::string_view piece) const override {
++- static absl::flat_hash_map<absl::string_view, int,
++- string_util::string_view_hash>
++- kMap = {{"<unk>", 0}, {"<s>", 1}, {"</s>", 2}, {WS "ABC", 3},
++- {WS "DE", 4}, {"F", 5}, {"G" WS "H", 6}, {WS, 7}};
+++ static absl::flat_hash_map<absl::string_view, int> kMap = {
+++ {"<unk>", 0}, {"<s>", 1}, {"</s>", 2}, {WS "ABC", 3},
+++ {WS "DE", 4}, {"F", 5}, {"G" WS "H", 6}, {WS, 7}};
++ return port::FindWithDefault(kMap, piece, 0);
++ }
++
++@@ -1058,18 +1057,6 @@ TEST(SentencePieceProcessorTest, EndToEndTest) {
++ EXPECT_EQ(2, sp.eos_id());
++ EXPECT_EQ(-1, sp.pad_id());
++
++- {
++- // Verify the default encoder version.
++- EXPECT_EQ(EncoderVersion::kOptimized, sp.GetEncoderVersion());
++-
++- // Set the encoder version to original and verify.
++- EXPECT_TRUE(sp.SetEncoderVersion(EncoderVersion::kOriginal).ok());
++- EXPECT_EQ(EncoderVersion::kOriginal, sp.GetEncoderVersion());
++-
++- // Set back to the default encoder version.
++- EXPECT_TRUE(sp.SetEncoderVersion(EncoderVersion::kOptimized).ok());
++- }
++-
++ {
++ std::vector<std::string> sps;
++ const std::vector<std::string> expected_str = {WS, "ab", "c"};
++@@ -1574,4 +1561,77 @@ TEST(SentencePieceProcessorTest, VocabularyTest) {
++ EXPECT_FALSE(sp.IsUnused(6));
++ EXPECT_FALSE(sp.IsUnused(7));
++ }
+++
+++TEST(SentencePieceProcessorTest, ImmutableSentencePieceTextTest) {
+++ ImmutableSentencePieceText spt;
+++ auto *v = spt.mutable_proto();
+++
+++ v->set_text("hello world");
+++ v->set_score(1.0);
+++ for (int i = 0; i < 10; ++i) {
+++ auto *p = v->add_pieces();
+++ p->set_surface(absl::StrCat("surface_", i));
+++ p->set_piece(absl::StrCat("surface_", i));
+++ p->set_id(i);
+++ p->set_begin(i + 10);
+++ p->set_end(i + 20);
+++ }
+++
+++ EXPECT_EQ(v->pieces_size(), spt.pieces_size());
+++ for (int i = 0; i < spt.pieces_size(); ++i) {
+++ EXPECT_EQ(v->pieces(i).surface(), spt.pieces(i).surface());
+++ EXPECT_EQ(v->pieces(i).piece(), spt.pieces(i).piece());
+++ EXPECT_EQ(v->pieces(i).id(), spt.pieces(i).id());
+++ EXPECT_EQ(v->pieces(i).begin(), spt.pieces(i).begin());
+++ EXPECT_EQ(v->pieces(i).end(), spt.pieces(i).end());
+++ }
+++
+++ int n = 0;
+++ for (auto &p : spt.pieces()) {
+++ EXPECT_EQ(v->pieces(n).surface(), p.surface());
+++ EXPECT_EQ(v->pieces(n).piece(), p.piece());
+++ EXPECT_EQ(v->pieces(n).id(), p.id());
+++ EXPECT_EQ(v->pieces(n).begin(), p.begin());
+++ EXPECT_EQ(v->pieces(n).end(), p.end());
+++ ++n;
+++ }
+++
+++ EXPECT_EQ(v->text(), spt.text());
+++ EXPECT_EQ(v->score(), spt.score());
+++ EXPECT_EQ(v->SerializeAsString(), spt.SerializeAsString());
+++
+++ // test copy.
+++ auto spt2 = spt;
+++ EXPECT_EQ(spt2.pieces_size(), spt.pieces_size());
+++ for (int i = 0; i < spt.pieces_size(); ++i) {
+++ EXPECT_EQ(spt2.pieces(i).surface(), spt.pieces(i).surface());
+++ EXPECT_EQ(spt2.pieces(i).piece(), spt.pieces(i).piece());
+++ EXPECT_EQ(spt2.pieces(i).id(), spt.pieces(i).id());
+++ EXPECT_EQ(spt2.pieces(i).begin(), spt.pieces(i).begin());
+++ EXPECT_EQ(spt2.pieces(i).end(), spt.pieces(i).end());
+++ }
+++}
+++
+++TEST(SentencePieceProcessorTest, ImmutableNBestSentencePieceTextTest) {
+++ ImmutableNBestSentencePieceText spt;
+++ auto *v = spt.mutable_proto();
+++ for (int i = 0; i < 10; ++i) {
+++ auto *p = v->add_nbests();
+++ p->set_text(absl::StrCat("text_", i));
+++ p->set_score(2.0 * i);
+++ }
+++
+++ EXPECT_EQ(v->nbests_size(), spt.nbests_size());
+++ for (int i = 0; i < v->nbests_size(); ++i) {
+++ EXPECT_EQ(v->nbests(i).text(), spt.nbests(i).text());
+++ EXPECT_EQ(v->nbests(i).score(), spt.nbests(i).score());
+++ }
+++ EXPECT_EQ(v->SerializeAsString(), spt.SerializeAsString());
+++
+++ // test copy.
+++ auto spt2 = spt;
+++ EXPECT_EQ(spt2.nbests_size(), spt.nbests_size());
+++ EXPECT_EQ(spt2.SerializeAsString(), spt.SerializeAsString());
+++}
+++
++ } // namespace sentencepiece
++diff --git a/src/unigram_model.cc b/src/unigram_model.cc
++index ea48912..d9f1ce9 100644
++--- a/src/unigram_model.cc
+++++ b/src/unigram_model.cc
++@@ -198,16 +198,17 @@ Lattice::LatticePathWithScore Lattice::Viterbi() {
++ return retval;
++ }
++
++-std::vector<float> Lattice::ForwardAlgorithm(float theta) const {
+++std::vector<float> Lattice::ForwardAlgorithm(float inv_theta) const {
++ const int len = size();
++ std::vector<float> alpha(node_allocator_.size(), 0.0);
++
++ for (int pos = 0; pos <= len; ++pos) {
++ for (Node *rnode : begin_nodes_[pos]) {
++ for (Node *lnode : end_nodes_[pos]) {
++- alpha[rnode->node_id] = LogSumExp(
++- alpha[rnode->node_id], theta * lnode->score + alpha[lnode->node_id],
++- lnode == end_nodes_[pos][0]);
+++ alpha[rnode->node_id] =
+++ LogSumExp(alpha[rnode->node_id],
+++ inv_theta * lnode->score + alpha[lnode->node_id],
+++ lnode == end_nodes_[pos][0]);
++ }
++ }
++ }
++@@ -215,7 +216,7 @@ std::vector<float> Lattice::ForwardAlgorithm(float theta) const {
++ return alpha;
++ }
++
++-std::vector<float> Lattice::BackwardAlgorithm(float theta) const {
+++std::vector<float> Lattice::BackwardAlgorithm(float inv_theta) const {
++ const int len = size();
++ std::vector<float> beta(node_allocator_.size(), 0.0);
++
++@@ -260,17 +261,16 @@ float Lattice::PopulateMarginal(float freq,
++ return freq * Z;
++ }
++
++-float Lattice::CalculateEntropy(float theta) const {
+++float Lattice::CalculateEntropy(float inv_theta) const {
++ const int len = size();
++
++ // alpha[node_id] is the marginal prob of sequence up to start of node
++ // H is entropy of sequence
++ // the index of alpha/H is Node::node_id.
++- std::vector<float> alpha(node_allocator_.size(), 0.0);
++ std::vector<float> H(node_allocator_.size(), 0.0);
++
++ // Populate the forward marginals to get the normalising constant
++- alpha = ForwardAlgorithm(theta);
+++ const auto alpha = ForwardAlgorithm(inv_theta);
++
++ // Now populate the forward entropies
++ for (int pos = 0; pos <= len; ++pos) {
++@@ -280,7 +280,7 @@ float Lattice::CalculateEntropy(float theta) const {
++
++ // We have to normalise p(lnode) by the marginal contribution it makes
++ const float lnode_transition_prob =
++- ((theta * lnode->score) + alpha[lnode->node_id] -
+++ ((inv_theta * lnode->score) + alpha[lnode->node_id] -
++ alpha[rnode->node_id]);
++ H[rnode->node_id] += std::exp(lnode_transition_prob) *
++ (H[lnode->node_id] + lnode_transition_prob);
++@@ -345,7 +345,7 @@ Hypothesis *CloneHypAndDependents(
++
++ std::vector<Lattice::LatticePathWithScore> Lattice::NBest(size_t nbest_size,
++ bool sample,
++- float theta) {
+++ float inv_theta) {
++ if (nbest_size < 1) {
++ LOG(WARNING) << "nbest_size >= 1. Returns empty result.";
++ return {};
++@@ -391,7 +391,7 @@ std::vector<Lattice::LatticePathWithScore> Lattice::NBest(size_t nbest_size,
++
++ if (sample) {
++ // Run forwards algorithm to get normalising constants
++- alpha = ForwardAlgorithm(theta);
+++ alpha = ForwardAlgorithm(inv_theta);
++ // f(eos) = Gumbel(0), as it is the perturbed score of the entire lattice.
++ eos->fx = Gumbel();
++ } else {
++@@ -432,7 +432,8 @@ std::vector<Lattice::LatticePathWithScore> Lattice::NBest(size_t nbest_size,
++ for (int i = 0; i < end_nodes(node->pos).size(); i++) {
++ Node *lnode = end_nodes(node->pos)[i];
++ // Calculate backwards transition score
++- probs[i] = top->gx + alpha[lnode->node_id] + (theta * lnode->score) - Z;
+++ probs[i] =
+++ top->gx + alpha[lnode->node_id] + (inv_theta * lnode->score) - Z;
++ perturbed_probs[i] = probs[i] + Gumbel();
++ if (perturbed_probs[i] > max_score) {
++ max_score = perturbed_probs[i];
++@@ -508,13 +509,13 @@ std::vector<Lattice::LatticePathWithScore> Lattice::NBest(size_t nbest_size,
++ return results;
++ }
++
++-std::vector<Lattice::Node *> Lattice::Sample(float theta) {
+++std::vector<Lattice::Node *> Lattice::Sample(float inv_theta) {
++ const int len = size();
++ if (len == 0) return {};
++
++ std::vector<float> alpha(node_allocator_.size(), 0.0);
++
++- alpha = ForwardAlgorithm(theta);
+++ alpha = ForwardAlgorithm(inv_theta);
++
++ auto *mt = random::GetRandomGenerator();
++
++@@ -526,8 +527,8 @@ std::vector<Lattice::Node *> Lattice::Sample(float theta) {
++ while (true) {
++ probs.clear();
++ for (const Node *lnode : end_nodes_[node->pos]) {
++- probs.push_back(std::exp(static_cast<double>(alpha[lnode->node_id] +
++- theta * lnode->score - Z)));
+++ probs.push_back(std::exp(static_cast<double>(
+++ alpha[lnode->node_id] + inv_theta * lnode->score - Z)));
++ }
++ std::discrete_distribution<int> dist(probs.begin(), probs.end());
++ node = end_nodes_[node->pos][dist(*mt)];
++@@ -721,7 +722,7 @@ NBestEncodeResult Model::NBestEncode(absl::string_view normalized,
++ }
++
++ EncodeResult Model::SampleEncode(absl::string_view normalized,
++- float theta) const {
+++ float inv_theta) const {
++ if (!status().ok() || normalized.empty()) {
++ return {};
++ }
++@@ -731,7 +732,7 @@ EncodeResult Model::SampleEncode(absl::string_view normalized,
++ PopulateNodes(&lattice);
++
++ EncodeResult results;
++- for (const auto *node : lattice.Sample(theta)) {
+++ for (const auto *node : lattice.Sample(inv_theta)) {
++ results.emplace_back(node->piece, node->id);
++ }
++
++@@ -739,7 +740,7 @@ EncodeResult Model::SampleEncode(absl::string_view normalized,
++ }
++
++ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++- float theta, int samples,
+++ float inv_theta, int samples,
++ bool wor,
++ bool include_best) const {
++ if (!status().ok() || normalized.empty()) {
++@@ -750,16 +751,16 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++ lattice.SetSentence(normalized);
++ PopulateNodes(&lattice);
++
++- std::vector<float> alpha = lattice.ForwardAlgorithm(theta);
++- float marginal = alpha[lattice.eos_node()->node_id];
+++ const std::vector<float> alpha = lattice.ForwardAlgorithm(inv_theta);
+++ const float marginal = alpha[lattice.eos_node()->node_id];
++
++ if (include_best) {
++ if (!wor) {
++- LOG(FATAL) << "include_best not supported for wor false";
+++ LOG(ERROR) << "include_best not supported for wor false";
+++ return {};
++ }
++ EncodeResult result;
++- Lattice::LatticePathWithScore best_path = lattice.Viterbi();
++-
+++ const auto best_path = lattice.Viterbi();
++ for (const auto *node : best_path.first) {
++ result.emplace_back(node->piece, node->id);
++ }
++@@ -770,8 +771,7 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++
++ if (wor) {
++ // Draw k+1 samples as we need perturbed score of k+1th element
++- std::vector<Lattice::LatticePathWithScore> nbest_samples =
++- lattice.NBest(samples + 1, true, theta);
+++ auto nbest_samples = lattice.NBest(samples + 1, true, inv_theta);
++
++ if (include_best) {
++ std::vector<std::vector<Lattice::Node *>> nbest_paths(
++@@ -780,14 +780,13 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++ nbest_paths[i] = nbest_samples[i].first;
++ }
++ // Remove the best result from the samples if necessary
++- Lattice::LatticePathWithScore best_path = lattice.Viterbi();
+++ const auto best_path = lattice.Viterbi();
++
++ const int index_of_best =
++ (std::find(nbest_paths.begin(), nbest_paths.end(), best_path.first) -
++ nbest_paths.begin());
++
++ if (index_of_best != nbest_samples.size()) {
++- LOG(INFO) << "removing best path from samples";
++ nbest_samples.erase(nbest_samples.begin() + index_of_best);
++ } else {
++ nbest_samples.pop_back();
++@@ -803,7 +802,7 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++ float score = 0.0;
++
++ for (const auto *node : nbest.first) {
++- score += (theta * node->score);
+++ score += (inv_theta * node->score);
++ result.emplace_back(node->piece, node->id);
++ }
++
++@@ -814,8 +813,8 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++ for (auto &it : results) {
++ // Only modify non best sample inclusion probabilities.
++ if (it.second != 0.0) {
++- double x = it.second - kappa;
++- double y = std::exp(x);
+++ const double x = it.second - kappa;
+++ const double y = std::exp(x);
++ double inclusion_prob;
++ if (x <= -10) {
++ // Series expansion of the log Gumbel survival function up to eps.
++@@ -835,10 +834,10 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++
++ float score = 0.0;
++ EncodeResult result;
++- std::vector<Lattice::Node *> sample = lattice.Sample(theta);
+++ const std::vector<Lattice::Node *> sample = lattice.Sample(inv_theta);
++ for (const auto *node : sample) {
++ result.emplace_back(node->piece, node->id);
++- score += (theta * node->score);
+++ score += (inv_theta * node->score);
++ }
++ results.emplace_back(result, score - marginal);
++ }
++@@ -847,12 +846,13 @@ NBestEncodeResult Model::SampleEncodeAndScore(absl::string_view normalized,
++ return results;
++ }
++
++-float Model::CalculateEntropy(absl::string_view normalized, float theta) const {
+++float Model::CalculateEntropy(absl::string_view normalized,
+++ float inv_theta) const {
++ Lattice lattice;
++ lattice.SetSentence(normalized);
++ PopulateNodes(&lattice);
++
++- return lattice.CalculateEntropy(theta);
+++ return lattice.CalculateEntropy(inv_theta);
++ }
++
++ bool Model::VerifyOutputsEquivalent(absl::string_view expected,
++diff --git a/src/unigram_model.h b/src/unigram_model.h
++index 448e489..aa4f28f 100644
++--- a/src/unigram_model.h
+++++ b/src/unigram_model.h
++@@ -173,6 +173,18 @@ class Model : public ModelInterface {
++ bool VerifyOutputsEquivalent(absl::string_view expected,
++ absl::string_view actual) const override;
++
+++ enum EncoderVersion {
+++ kOptimized, // The optimized encoder.
+++ kOriginal // The original encoder.
+++ };
+++
+++ void SetEncoderVersion(EncoderVersion encoder_version) {
+++ encoder_version_ = encoder_version;
+++ }
+++
+++ // Returns the current encoder version in use.
+++ EncoderVersion GetEncoderVersion() const { return encoder_version_; }
+++
++ protected:
++ // Builds a Trie index.
++ void BuildTrie(std::vector<std::pair<absl::string_view, int>> *pieces);
++@@ -195,6 +207,9 @@ class Model : public ModelInterface {
++ // Maximum size of the return value of Trie, which corresponds
++ // to the maximum size of shared common prefix in the sentence pieces.
++ int trie_results_size_;
+++
+++ // encoder version.
+++ EncoderVersion encoder_version_ = kOptimized;
++ };
++
++ } // namespace unigram
++diff --git a/src/unigram_model_test.cc b/src/unigram_model_test.cc
++index 8049d20..221bac2 100644
++--- a/src/unigram_model_test.cc
+++++ b/src/unigram_model_test.cc
++@@ -12,6 +12,8 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
+++#include "unigram_model.h"
+++
++ #include <cmath>
++ #include <map>
++ #include <string>
++@@ -22,7 +24,6 @@
++ #include "testharness.h"
++ #include "third_party/absl/strings/str_cat.h"
++ #include "third_party/absl/strings/str_join.h"
++-#include "unigram_model.h"
++ #include "util.h"
++
++ namespace sentencepiece {
++@@ -249,14 +250,14 @@ TEST(LatticeTest, NBestSampleTest) {
++
++ // Calculate expected probabilities of each path
++ // Note that sampling without replacement affects the expected frequencies!
++- const std::vector<double> kTheta = {0.0, 0.01, 0.5, 0.7, 1.0};
++- for (const auto theta : kTheta) {
+++ const std::vector<double> kInv_Theta = {0.0, 0.01, 0.5, 0.7, 1.0};
+++ for (const auto inv_theta : kInv_Theta) {
++ std::vector<std::string> strings = {"ABC", "AB C", "A BC", "A B C"};
++ std::map<std::string, float> probs;
++- probs["ABC"] = std::exp(theta * 1.0);
++- probs["AB C"] = std::exp(theta * (0.2 + 0.1));
++- probs["A BC"] = std::exp(theta * (0.0 + 0.5));
++- probs["A B C"] = std::exp(theta * (0.0 + 0.0 + 0.1));
+++ probs["ABC"] = std::exp(inv_theta * 1.0);
+++ probs["AB C"] = std::exp(inv_theta * (0.2 + 0.1));
+++ probs["A BC"] = std::exp(inv_theta * (0.0 + 0.5));
+++ probs["A B C"] = std::exp(inv_theta * (0.0 + 0.0 + 0.1));
++
++ for (const auto &it : strings) {
++ EXPECT_EQ(1, probs.count(it));
++@@ -298,7 +299,7 @@ TEST(LatticeTest, NBestSampleTest) {
++ for (const auto num_samples : kNumSamples) {
++ std::map<std::string, int> counts;
++ for (int i = 0; i < kTrials; i++) {
++- auto nbests = lattice.NBest(num_samples, true, theta);
+++ auto nbests = lattice.NBest(num_samples, true, inv_theta);
++ for (const auto &nbest : nbests) {
++ counts[GetTokenized(nbest.first)]++;
++ }
++@@ -329,14 +330,14 @@ TEST(LatticeTest, CalculateEntropyTest) {
++ InsertWithScore(&lattice, 0, 3, 1.0); // ABC
++
++ // Calculate expected probabilities of each path
++- const std::vector<double> kTheta = {0.0, 0.01, 0.5, 0.7, 1.0};
++- for (const auto theta : kTheta) {
+++ const std::vector<double> kInv_Theta = {0.0, 0.01, 0.5, 0.7, 1.0};
+++ for (const auto inv_theta : kInv_Theta) {
++ std::vector<std::string> strings = {"ABC", "AB C", "A BC", "A B C"};
++ std::map<std::string, float> probs;
++- probs["ABC"] = std::exp(theta * 1.0);
++- probs["AB C"] = std::exp(theta * (0.2 + 0.1));
++- probs["A BC"] = std::exp(theta * (0.0 + 0.5));
++- probs["A B C"] = std::exp(theta * (0.0 + 0.0 + 0.1));
+++ probs["ABC"] = std::exp(inv_theta * 1.0);
+++ probs["AB C"] = std::exp(inv_theta * (0.2 + 0.1));
+++ probs["A BC"] = std::exp(inv_theta * (0.0 + 0.5));
+++ probs["A B C"] = std::exp(inv_theta * (0.0 + 0.0 + 0.1));
++
++ double Z = 0.0;
++ for (const auto &it : probs) Z += it.second;
++@@ -349,7 +350,7 @@ TEST(LatticeTest, CalculateEntropyTest) {
++ for (const auto &it : probs) {
++ entropy += (it.second * std::log(it.second));
++ }
++- EXPECT_NEAR(-entropy, lattice.CalculateEntropy(theta), 0.02);
+++ EXPECT_NEAR(-entropy, lattice.CalculateEntropy(inv_theta), 0.02);
++ }
++ }
++
++@@ -364,9 +365,9 @@ TEST(LatticeTest, ForwardAlgorithmTest) {
++ InsertWithScore(&lattice, 1, 2, 0.5); // BC
++ InsertWithScore(&lattice, 0, 3, 1.0); // ABC
++
++- const std::vector<float> kTheta = {0.0, 0.01, 0.5, 0.7, 1.0};
++- for (const auto theta : kTheta) {
++- std::vector<float> alpha = lattice.ForwardAlgorithm(theta);
+++ const std::vector<float> kInv_Theta = {0.0, 0.01, 0.5, 0.7, 1.0};
+++ for (const auto inv_theta : kInv_Theta) {
+++ std::vector<float> alpha = lattice.ForwardAlgorithm(inv_theta);
++ EXPECT_EQ(alpha.size(), 8); // 6 nodes, plus BOS, EOS
++ // only alpha[C], alpha[EOS] have non-zero alpha
++ for (int i : {0, 1, 2, 3}) {
++@@ -374,14 +375,15 @@ TEST(LatticeTest, ForwardAlgorithmTest) {
++ if (i < 2) {
++ EXPECT_EQ(alpha[node->node_id], 0.0);
++ } else if (i == 2) {
++- float Z =
++- std::log(std::exp(theta * (0.0 + 0.0)) + std::exp(theta * 0.2));
+++ float Z = std::log(std::exp(inv_theta * (0.0 + 0.0)) +
+++ std::exp(inv_theta * 0.2));
++ EXPECT_EQ(alpha[node->node_id], Z);
++ } else if (i == 3) {
++- float Z = std::log(std::exp(theta * (0.0 + 0.0 + 0.1)) + // A + B + C
++- std::exp(theta * (0.2 + 0.1)) + // AB + C
++- std::exp(theta * (0.0 + 0.5)) + // A + BC
++- std::exp(theta * 1.0)); // ABC
+++ float Z =
+++ std::log(std::exp(inv_theta * (0.0 + 0.0 + 0.1)) + // A + B + C
+++ std::exp(inv_theta * (0.2 + 0.1)) + // AB + C
+++ std::exp(inv_theta * (0.0 + 0.5)) + // A + BC
+++ std::exp(inv_theta * 1.0)); // ABC
++ EXPECT_EQ(Z, alpha[node->node_id]);
++ }
++ }
++@@ -435,14 +437,14 @@ TEST(LatticeTest, SampleTest) {
++ InsertWithScoreAndId(&lattice, 1, 2, 1.7, 4); // BC
++ InsertWithScoreAndId(&lattice, 0, 3, 1.8, 5); // ABC
++
++- const std::vector<double> kTheta = {0.0, 0.01, 0.5, 0.7, 1.0};
++- for (int i = 0; i < kTheta.size(); ++i) {
+++ const std::vector<double> kInv_Theta = {0.0, 0.01, 0.5, 0.7, 1.0};
+++ for (int i = 0; i < kInv_Theta.size(); ++i) {
++ std::map<std::string, double> probs;
++ // Expands all paths in the lattice.
++- probs["A B C"] = exp(kTheta[i] * (1.0 + 1.2 + 1.5)); // A B C
++- probs["AB C"] = exp(kTheta[i] * (1.6 + 1.5)); // AB C
++- probs["A BC"] = exp(kTheta[i] * (1.0 + 1.7)); // A BC
++- probs["ABC"] = exp(kTheta[i] * 1.8); // ABC
+++ probs["A B C"] = exp(kInv_Theta[i] * (1.0 + 1.2 + 1.5)); // A B C
+++ probs["AB C"] = exp(kInv_Theta[i] * (1.6 + 1.5)); // AB C
+++ probs["A BC"] = exp(kInv_Theta[i] * (1.0 + 1.7)); // A BC
+++ probs["ABC"] = exp(kInv_Theta[i] * 1.8); // ABC
++
++ // Computes expected probabilities.
++ double Z = 0.0;
++@@ -453,7 +455,7 @@ TEST(LatticeTest, SampleTest) {
++ constexpr int kTrial = 100000;
++ std::map<std::string, int> freq;
++ for (int n = 0; n < kTrial; ++n) {
++- freq[GetTokenized(lattice.Sample(kTheta[i]))]++;
+++ freq[GetTokenized(lattice.Sample(kInv_Theta[i]))]++;
++ }
++
++ EXPECT_EQ(probs.size(), freq.size());
++@@ -480,18 +482,18 @@ ModelProto MakeBaseModelProto() {
++ }
++
++ // Returns model protos in parameterized tests.
++-const std::vector<EncoderVersion> &GetEncoderVersions() {
++- static const std::vector<EncoderVersion> &v =
++- *new std::vector<EncoderVersion>{EncoderVersion::kOptimized,
++- EncoderVersion::kOriginal};
+++const std::vector<Model::EncoderVersion> &GetEncoderVersions() {
+++ static const std::vector<Model::EncoderVersion> &v =
+++ *new std::vector<Model::EncoderVersion>{Model::kOptimized,
+++ Model::kOriginal};
++ return v;
++ }
++
++-class UnigramModelTest : public test::TestWithParam<EncoderVersion> {
+++class UnigramModelTest : public test::TestWithParam<Model::EncoderVersion> {
++ protected:
++ void SetUp() override { encoder_version_ = GetParam(); }
++ void TearDown() override {}
++- EncoderVersion encoder_version_;
+++ Model::EncoderVersion encoder_version_;
++ };
++
++ void AddPiece(ModelProto *model_proto, const std::string &piece,
++@@ -530,15 +532,15 @@ TEST(UnigramModelTest, SampleEncodeAndScoreTest) {
++ lattice.SetSentence("ABC");
++ model.PopulateNodes(&lattice);
++
++- std::vector<float> kTheta = {0.0, 1.0};
+++ std::vector<float> kInv_Theta = {0.0, 1.0};
++
++- for (const auto theta : kTheta) {
+++ for (const auto inv_theta : kInv_Theta) {
++ std::vector<std::string> strings = {"ABC", "AB C", "A BC", "A B C"};
++ std::map<std::string, float> probs;
++- probs["ABC"] = std::exp(theta * 1.0);
++- probs["AB C"] = std::exp(theta * (0.2 + 0.1));
++- probs["A BC"] = std::exp(theta * (0.0 + 0.5));
++- probs["A B C"] = std::exp(theta * (0.0 + 0.0 + 0.1));
+++ probs["ABC"] = std::exp(inv_theta * 1.0);
+++ probs["AB C"] = std::exp(inv_theta * (0.2 + 0.1));
+++ probs["A BC"] = std::exp(inv_theta * (0.0 + 0.5));
+++ probs["A B C"] = std::exp(inv_theta * (0.0 + 0.0 + 0.1));
++
++ for (const auto &it : strings) {
++ EXPECT_EQ(1, probs.count(it));
++@@ -579,8 +581,8 @@ TEST(UnigramModelTest, SampleEncodeAndScoreTest) {
++ std::map<std::string, float> scores;
++ int kTrials = 50000;
++ for (int i = 0; i < kTrials; i++) {
++- NBestEncodeResult sample =
++- model.SampleEncodeAndScore("ABC", theta, num_samples, true, false);
+++ NBestEncodeResult sample = model.SampleEncodeAndScore(
+++ "ABC", inv_theta, num_samples, true, false);
++
++ for (const auto &it : sample) {
++ std::vector<std::string> tokens;
++@@ -619,7 +621,7 @@ TEST_P(UnigramModelTest, PieceToIdTest) {
++ AddPiece(&model_proto, "d", 0.4);
++
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++
++ EXPECT_EQ(model_proto.SerializeAsString(),
++ model.model_proto().SerializeAsString());
++@@ -677,7 +679,7 @@ TEST_P(UnigramModelTest, PopulateNodesAllUnknownsTest) {
++ ModelProto model_proto = MakeBaseModelProto();
++ AddPiece(&model_proto, "x");
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++
++ Lattice lattice;
++ lattice.SetSentence("abc");
++@@ -701,7 +703,7 @@ TEST_P(UnigramModelTest, PopulateNodesTest) {
++ AddPiece(&model_proto, "bc", 0.4); // 6
++
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++
++ Lattice lattice;
++ lattice.SetSentence("abc");
++@@ -736,7 +738,7 @@ TEST_P(UnigramModelTest, PopulateNodesWithUnusedTest) {
++ model_proto.mutable_pieces(6)->set_type(ModelProto::SentencePiece::UNUSED);
++
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++
++ Lattice lattice;
++ lattice.SetSentence("abc");
++@@ -761,7 +763,7 @@ TEST_P(UnigramModelTest, ModelNBestTest) {
++ AddPiece(&model_proto, "abc", 10.0); // 8
++
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++
++ auto nbest = model.NBestEncode("", 10);
++ EXPECT_EQ(1, nbest.size());
++@@ -800,7 +802,7 @@ TEST_P(UnigramModelTest, EncodeTest) {
++ ModelProto::SentencePiece::USER_DEFINED);
++
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++
++ EncodeResult result;
++
++@@ -883,7 +885,7 @@ TEST_P(UnigramModelTest, EncodeWithUnusedTest) {
++ // No unused.
++ {
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++ const auto result = model.Encode("abcd");
++ EXPECT_EQ(1, result.size());
++ EXPECT_EQ("abcd", result[0].first);
++@@ -892,7 +894,7 @@ TEST_P(UnigramModelTest, EncodeWithUnusedTest) {
++ {
++ model_proto.mutable_pieces(3)->set_type(ModelProto::SentencePiece::UNUSED);
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++ const auto result = model.Encode("abcd");
++ EXPECT_EQ(2, result.size());
++ EXPECT_EQ("abc", result[0].first);
++@@ -903,7 +905,7 @@ TEST_P(UnigramModelTest, EncodeWithUnusedTest) {
++ model_proto.mutable_pieces(3)->set_type(ModelProto::SentencePiece::UNUSED);
++ model_proto.mutable_pieces(5)->set_type(ModelProto::SentencePiece::UNUSED);
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++ const auto result = model.Encode("abcd");
++ EXPECT_EQ(2, result.size());
++ EXPECT_EQ("abc", result[0].first);
++@@ -917,7 +919,7 @@ TEST_P(UnigramModelTest, EncodeWithUnusedTest) {
++ model_proto.mutable_pieces(4)->set_type(ModelProto::SentencePiece::UNUSED);
++ model_proto.mutable_pieces(5)->set_type(ModelProto::SentencePiece::NORMAL);
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++ const auto result = model.Encode("abcd");
++ EXPECT_EQ(2, result.size());
++ EXPECT_EQ("ab", result[0].first);
++@@ -937,7 +939,7 @@ TEST_P(UnigramModelTest, VerifyOutputsEquivalent) {
++ AddPiece(&model_proto, "c", 2.0); // 9
++ AddPiece(&model_proto, "d", 1.0); // 10
++ Model model(model_proto);
++- EXPECT_TRUE(model.SetEncoderVersion(encoder_version_).ok());
+++ model.SetEncoderVersion(encoder_version_);
++ // Equivalent outputs.
++ EXPECT_TRUE(model.VerifyOutputsEquivalent("", ""));
++ EXPECT_TRUE(model.VerifyOutputsEquivalent("a b", "a b"));
++diff --git a/src/util.h b/src/util.h
++index 285676d..fb312f1 100644
++--- a/src/util.h
+++++ b/src/util.h
++@@ -60,17 +60,6 @@ uint32 GetRandomGeneratorSeed();
++ // String utilities
++ namespace string_util {
++
++-struct string_view_hash {
++- // DJB hash function.
++- inline size_t operator()(const absl::string_view &sp) const {
++- size_t hash = 5381;
++- for (size_t i = 0; i < sp.size(); ++i) {
++- hash = ((hash << 5) + hash) + sp[i];
++- }
++- return hash;
++- }
++-};
++-
++ template <typename Target>
++ inline bool lexical_cast(absl::string_view arg, Target *result) {
++ std::stringstream ss;
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Mon, 20 Jun 2022 01:35:11 +0900
++Subject: add verbose option
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ .github/workflows/cmake.yml | 2 +-
++ src/common.h | 13 -------------
++ src/normalizer.cc | 7 +++----
++ src/sentencepiece_processor.cc | 10 ++++++----
++ src/sentencepiece_processor.h | 9 +++------
++ src/util.h | 1 -
++ 6 files changed, 13 insertions(+), 29 deletions(-)
++
++diff --git a/.github/workflows/cmake.yml b/.github/workflows/cmake.yml
++index 7f19083..5108074 100644
++--- a/.github/workflows/cmake.yml
+++++ b/.github/workflows/cmake.yml
++@@ -45,7 +45,7 @@ jobs:
++
++ - name: Test
++ working-directory: ${{github.workspace}}/build
++- run: ctest -C Release
+++ run: ctest -C Release --output-on-failure
++
++ - name: Package
++ working-directory: ${{github.workspace}}/build
++diff --git a/src/common.h b/src/common.h
++index c27c352..ba951d6 100644
++--- a/src/common.h
+++++ b/src/common.h
++@@ -98,15 +98,6 @@ class Die {
++ private:
++ bool die_;
++ };
++-
++-template <typename T>
++-T &&CheckNotNull(const char *file, int line, const char *exprtext, T &&t) {
++- if (t == nullptr) {
++- std::cerr << file << "(" << line << ") " << exprtext;
++- Abort();
++- }
++- return std::forward<T>(t);
++-}
++ } // namespace error
++
++ namespace logging {
++@@ -158,10 +149,6 @@ inline const char *BaseName(const char *path) {
++ #define CHECK_LE(a, b) CHECK((a) <= (b))
++ #define CHECK_GT(a, b) CHECK((a) > (b))
++ #define CHECK_LT(a, b) CHECK((a) < (b))
++-#define CHECK_NOTNULL(val) \
++- ::sentencepiece::error::CheckNotNull( \
++- ::sentencepiece::logging::BaseName(__FILE__), __LINE__, \
++- "'" #val "' Must be non NULL", (val))
++
++ #define FRIEND_TEST(a, b) friend class a##_Test_##b;
++
++diff --git a/src/normalizer.cc b/src/normalizer.cc
++index d87f89b..2ab8084 100644
++--- a/src/normalizer.cc
+++++ b/src/normalizer.cc
++@@ -12,11 +12,12 @@
++ // See the License for the specific language governing permissions and
++ // limitations under the License.!
++
+++#include "normalizer.h"
+++
++ #include <utility>
++ #include <vector>
++
++ #include "common.h"
++-#include "normalizer.h"
++ #include "third_party/absl/memory/memory.h"
++ #include "third_party/absl/strings/match.h"
++ #include "third_party/absl/strings/string_view.h"
++@@ -46,9 +47,7 @@ Normalizer::~Normalizer() {}
++
++ void Normalizer::Init() {
++ absl::string_view index = spec_->precompiled_charsmap();
++- if (index.empty()) {
++- LOG(INFO) << "precompiled_charsmap is empty. use identity normalization.";
++- } else {
+++ if (!index.empty()) {
++ absl::string_view trie_blob, normalized;
++ #ifdef IS_BIG_ENDIAN
++ status_ = DecodePrecompiledCharsMap(index, &trie_blob, &normalized,
++diff --git a/src/sentencepiece_processor.cc b/src/sentencepiece_processor.cc
++index a6f5395..805e0f9 100644
++--- a/src/sentencepiece_processor.cc
+++++ b/src/sentencepiece_processor.cc
++@@ -67,12 +67,12 @@ ImmutableSentencePieceText::ImmutableSentencePiece::ImmutableSentencePiece(
++ const SentencePieceText_SentencePiece &sp)
++ : sp_(&sp) {}
++
++-absl::string_view ImmutableSentencePieceText::ImmutableSentencePiece::piece()
+++const std::string &ImmutableSentencePieceText::ImmutableSentencePiece::piece()
++ const {
++ return sp_->piece();
++ }
++
++-absl::string_view ImmutableSentencePieceText::ImmutableSentencePiece::surface()
+++const std::string &ImmutableSentencePieceText::ImmutableSentencePiece::surface()
++ const {
++ return sp_->surface();
++ }
++@@ -109,8 +109,10 @@ ImmutableSentencePieceText::pieces(int index) const {
++ spt_->pieces(index));
++ }
++
++-absl::string_view ImmutableSentencePieceText::text() const {
++- return spt_ ? spt_->text() : "";
+++const std::string &ImmutableSentencePieceText::text() const {
+++ if (spt_) return spt_->text();
+++ static std::string *kEmptyString = new std::string();
+++ return *kEmptyString;
++ }
++
++ float ImmutableSentencePieceText::score() const {
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index 51c5b3b..8124c59 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -165,8 +165,8 @@ class ImmutableSentencePieceText {
++ class ImmutableSentencePiece {
++ public:
++ ~ImmutableSentencePiece() = default;
++- absl::string_view piece() const;
++- absl::string_view surface() const;
+++ const std::string &piece() const;
+++ const std::string &surface() const;
++ uint32_t id() const;
++ uint32_t begin() const;
++ uint32_t end() const;
++@@ -182,7 +182,7 @@ class ImmutableSentencePieceText {
++ std::vector<ImmutableSentencePiece> pieces() const;
++ size_t pieces_size() const;
++ ImmutableSentencePiece pieces(int index) const;
++- absl::string_view text() const;
+++ const std::string &text() const;
++ float score() const;
++
++ std::string SerializeAsString() const;
++@@ -193,7 +193,6 @@ class ImmutableSentencePieceText {
++ SentencePieceText *mutable_proto();
++
++ friend class ImmutableNBestSentencePieceText;
++- friend class SentencePieceProcessor;
++
++ private:
++ explicit ImmutableSentencePieceText(const SentencePieceText &spt);
++@@ -222,8 +221,6 @@ class ImmutableNBestSentencePieceText {
++ // it returns the raw pointer managed by the shared_ptr.
++ NBestSentencePieceText *mutable_proto();
++
++- friend class SentencePieceProcessor;
++-
++ private:
++ std::shared_ptr<NBestSentencePieceText> rep_;
++ };
++diff --git a/src/util.h b/src/util.h
++index fb312f1..01a561f 100644
++--- a/src/util.h
+++++ b/src/util.h
++@@ -94,7 +94,6 @@ inline bool lexical_cast(absl::string_view arg, std::string *result) {
++
++ template <typename T>
++ inline bool DecodePOD(absl::string_view str, T *result) {
++- CHECK_NOTNULL(result);
++ if (sizeof(*result) != str.size()) {
++ return false;
++ }
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Mon, 1 Aug 2022 17:19:09 +0900
++Subject: Supports ImmutableSentencePieceText from python module
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/__init__.py | 228 ++-
++ python/src/sentencepiece/sentencepiece.i | 310 ++-
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 2310 ++++++++++++++++++-----
++ python/test/sentencepiece_test.py | 62 +-
++ src/sentencepiece_processor.cc | 87 +-
++ src/sentencepiece_processor.h | 61 +-
++ src/sentencepiece_processor_test.cc | 137 +-
++ 7 files changed, 2524 insertions(+), 671 deletions(-)
++
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index 1543d32..69a9825 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -61,6 +61,98 @@ class _SwigNonDynamicMeta(type):
++ __setattr__ = _swig_setattr_nondynamic_class_variable(type.__setattr__)
++
++
+++class ImmutableSentencePieceText_ImmutableSentencePiece(object):
+++ thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
+++ __repr__ = _swig_repr
+++
+++ def __init__(self):
+++ _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_swiginit(self, _sentencepiece.new_ImmutableSentencePieceText_ImmutableSentencePiece())
+++ __swig_destroy__ = _sentencepiece.delete_ImmutableSentencePieceText_ImmutableSentencePiece
+++
+++ def piece(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_piece(self)
+++
+++ def surface(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_surface(self)
+++
+++ def id(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_id(self)
+++
+++ def begin(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_begin(self)
+++
+++ def end(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_end(self)
+++
+++# Register ImmutableSentencePieceText_ImmutableSentencePiece in _sentencepiece:
+++_sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_swigregister(ImmutableSentencePieceText_ImmutableSentencePiece)
+++
+++class ImmutableSentencePieceText(object):
+++ thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
+++ __repr__ = _swig_repr
+++
+++ def __init__(self):
+++ _sentencepiece.ImmutableSentencePieceText_swiginit(self, _sentencepiece.new_ImmutableSentencePieceText())
+++ __swig_destroy__ = _sentencepiece.delete_ImmutableSentencePieceText
+++
+++ def pieces_size(self):
+++ return _sentencepiece.ImmutableSentencePieceText_pieces_size(self)
+++
+++ def text(self):
+++ return _sentencepiece.ImmutableSentencePieceText_text(self)
+++
+++ def score(self):
+++ return _sentencepiece.ImmutableSentencePieceText_score(self)
+++
+++ def SerializeAsString(self):
+++ return _sentencepiece.ImmutableSentencePieceText_SerializeAsString(self)
+++
+++ def pieces(self, index):
+++ return _sentencepiece.ImmutableSentencePieceText_pieces(self, index)
+++
+++ def __len__(self):
+++ return self.pieces_size()
+++
+++ def __getitem__(self, i):
+++ return self.pieces(i)
+++
+++ def __eq__(self, other):
+++ return self.SerializeAsString() == other.SerializeAsString()
+++
+++
+++# Register ImmutableSentencePieceText in _sentencepiece:
+++_sentencepiece.ImmutableSentencePieceText_swigregister(ImmutableSentencePieceText)
+++
+++class ImmutableNBestSentencePieceText(object):
+++ thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
+++ __repr__ = _swig_repr
+++
+++ def __init__(self):
+++ _sentencepiece.ImmutableNBestSentencePieceText_swiginit(self, _sentencepiece.new_ImmutableNBestSentencePieceText())
+++ __swig_destroy__ = _sentencepiece.delete_ImmutableNBestSentencePieceText
+++
+++ def nbests_size(self):
+++ return _sentencepiece.ImmutableNBestSentencePieceText_nbests_size(self)
+++
+++ def SerializeAsString(self):
+++ return _sentencepiece.ImmutableNBestSentencePieceText_SerializeAsString(self)
+++
+++ def nbests(self, index):
+++ return _sentencepiece.ImmutableNBestSentencePieceText_nbests(self, index)
+++
+++ def __len__(self):
+++ return self.nbests_size()
+++
+++ def __getitem__(self, i):
+++ return self.nbests(i)
+++
+++ def __eq__(self, other):
+++ return self.SerializeAsString() == other.SerializeAsString()
+++
+++
+++# Register ImmutableNBestSentencePieceText in _sentencepiece:
+++_sentencepiece.ImmutableNBestSentencePieceText_swigregister(ImmutableNBestSentencePieceText)
+++
++ class SentencePieceProcessor(object):
++ thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
++ __repr__ = _swig_repr
++@@ -87,12 +179,6 @@ class SentencePieceProcessor(object):
++ def LoadVocabulary(self, filename, threshold):
++ return _sentencepiece.SentencePieceProcessor_LoadVocabulary(self, filename, threshold)
++
++- def SampleEncodeAndScoreAsPieces(self, input, num_samples, theta, wor, include_best):
++- return _sentencepiece.SentencePieceProcessor_SampleEncodeAndScoreAsPieces(self, input, num_samples, theta, wor, include_best)
++-
++- def SampleEncodeAndScoreAsIds(self, input, num_samples, theta, wor, include_best):
++- return _sentencepiece.SentencePieceProcessor_SampleEncodeAndScoreAsIds(self, input, num_samples, theta, wor, include_best)
++-
++ def CalculateEntropy(self, *args):
++ return _sentencepiece.SentencePieceProcessor_CalculateEntropy(self, *args)
++
++@@ -147,6 +233,9 @@ class SentencePieceProcessor(object):
++ def _EncodeAsSerializedProto(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__EncodeAsSerializedProto(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
+++ def _EncodeAsImmutableProto(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsImmutableProto(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
++ def _EncodeAsIdsBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__EncodeAsIdsBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
++@@ -156,6 +245,9 @@ class SentencePieceProcessor(object):
++ def _EncodeAsSerializedProtoBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__EncodeAsSerializedProtoBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
+++ def _EncodeAsImmutableProtoBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__EncodeAsImmutableProtoBatch(self, ins, num_threads, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++
++ def _DecodeIds(self, ids):
++ return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
++
++@@ -168,6 +260,12 @@ class SentencePieceProcessor(object):
++ def _DecodePiecesAsSerializedProto(self, pieces):
++ return _sentencepiece.SentencePieceProcessor__DecodePiecesAsSerializedProto(self, pieces)
++
+++ def _DecodeIdsAsImmutableProto(self, ids):
+++ return _sentencepiece.SentencePieceProcessor__DecodeIdsAsImmutableProto(self, ids)
+++
+++ def _DecodePiecesAsImmutableProto(self, pieces):
+++ return _sentencepiece.SentencePieceProcessor__DecodePiecesAsImmutableProto(self, pieces)
+++
++ def _DecodeIdsBatch(self, ins, num_threads):
++ return _sentencepiece.SentencePieceProcessor__DecodeIdsBatch(self, ins, num_threads)
++
++@@ -180,6 +278,9 @@ class SentencePieceProcessor(object):
++ def _DecodePiecesAsSerializedProtoBatch(self, ins, num_threads):
++ return _sentencepiece.SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch(self, ins, num_threads)
++
+++ def _DecodePiecesAsImmutableProtoBatch(self, ins, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch(self, ins, num_threads)
+++
++ def _NBestEncodeAsIds(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__NBestEncodeAsIds(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
++
++@@ -189,17 +290,26 @@ class SentencePieceProcessor(object):
++ def _NBestEncodeAsSerializedProto(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece):
++ return _sentencepiece.SentencePieceProcessor__NBestEncodeAsSerializedProto(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
++
++- def _SampleEncodeAndScoreAsIds(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
++- return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsIds(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
+++ def _NBestEncodeAsImmutableProto(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__NBestEncodeAsImmutableProto(self, text, nbest_size, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _SampleEncodeAndScoreAsIds(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsIds(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
++
++- def _SampleEncodeAndScoreAsPieces(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
++- return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsPieces(self, text, num_samples, theta, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
+++ def _SampleEncodeAndScoreAsPieces(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsPieces(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
++
++- def _CalculateEntropy(self, text, theta):
++- return _sentencepiece.SentencePieceProcessor__CalculateEntropy(self, text, theta)
+++ def _SampleEncodeAndScoreAsSerializedProto(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
++
++- def _CalculateEntropyBatch(self, ins, theta, num_threads):
++- return _sentencepiece.SentencePieceProcessor__CalculateEntropyBatch(self, ins, theta, num_threads)
+++ def _SampleEncodeAndScoreAsImmutableProto(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece):
+++ return _sentencepiece.SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto(self, text, num_samples, alpha, wor, include_best, add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ def _CalculateEntropy(self, text, alpha):
+++ return _sentencepiece.SentencePieceProcessor__CalculateEntropy(self, text, alpha)
+++
+++ def _CalculateEntropyBatch(self, ins, alpha, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__CalculateEntropyBatch(self, ins, alpha, num_threads)
++
++ def Init(self,
++ model_file=None,
++@@ -319,9 +429,12 @@ class SentencePieceProcessor(object):
++ if out_type is str:
++ return self._EncodeAsPiecesBatch(input, num_threads, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto' or out_type == 'proto':
++ return self._EncodeAsSerializedProtoBatch(input, num_threads, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'immutable_proto':
+++ return self._EncodeAsImmutableProtoBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
++ if out_type is int:
++ return self._EncodeAsIds(input, enable_sampling, nbest_size,
++@@ -329,9 +442,12 @@ class SentencePieceProcessor(object):
++ if out_type is str:
++ return self._EncodeAsPieces(input, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto' or out_type == 'proto':
++ return self._EncodeAsSerializedProto(input, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'immutable_proto':
+++ return self._EncodeAsImmutableProto(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
++ raise RuntimeError('unknown out_type={}'.format(out_type))
++ return None
++@@ -346,7 +462,11 @@ class SentencePieceProcessor(object):
++
++
++ def EncodeAsSerializedProto(self, input, **kwargs):
++- return self.Encode(input=input, out_type='proto', **kwargs)
+++ return self.Encode(input=input, out_type='serialized_proto', **kwargs)
+++
+++
+++ def EncodeAsImmutableProto(self, input, **kwargs):
+++ return self.Encode(input=input, out_type='immutable_proto', **kwargs)
++
++
++ def SampleEncodeAsPieces(self, input, nbest_size=None, alpha=None, **kwargs):
++@@ -361,7 +481,12 @@ class SentencePieceProcessor(object):
++
++ def SampleEncodeAsSerializedProto(self, input, nbest_size=None, alpha=None, **kwargs):
++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
++- out_type='proto', enable_sampling=True, **kwargs)
+++ out_type='serialized_proto', enable_sampling=True, **kwargs)
+++
+++
+++ def SampleEncodeAsImmutableProto(self, input, nbest_size=None, alpha=None, **kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type='immutable_proto', enable_sampling=True, **kwargs)
++
++
++ def NBestEncode(self,
++@@ -407,9 +532,12 @@ class SentencePieceProcessor(object):
++ if out_type is str:
++ return self._NBestEncodeAsPieces(text, nbest_size,
++ add_bos, add_eos, reverse, emit_unk_piece)
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto' or out_type == 'proto':
++ return self._NBestEncodeAsSerializedProto(text, nbest_size,
++ add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'immutable_proto':
+++ return self._NBestEncodeAsImmutableProto(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++@@ -429,7 +557,12 @@ class SentencePieceProcessor(object):
++
++ def NBestEncodeAsSerializedProto(self, input, nbest_size=None, **kwargs):
++ return self.NBestEncode(input=input, nbest_size=nbest_size,
++- out_type='proto', **kwargs)
+++ out_type='serialized_proto', **kwargs)
+++
+++
+++ def NBestEncodeAsImmutableProto(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type='immutable_proto', **kwargs)
++
++
++ def SampleEncodeAndScore(self,
++@@ -440,20 +573,20 @@ class SentencePieceProcessor(object):
++ reverse=None,
++ emit_unk_piece=None,
++ num_samples=None,
++- theta=None,
+++ alpha=None,
++ wor=None,
++ include_best=None):
++ """SampleEncodeAndScore text input to segmented ids or tokens.
++
++ Args:
++ input: input string. accepsts list of string.
++- out_type: output type. int or str or 'proto'.
+++ out_type: output type. int or str or 'serialized_proto' or 'immutable_proto'
++ add_bos: Add <s> to the result (Default = false)
++ add_eos: Add </s> to the result (Default = false) <s>/</s> is added after reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++ emit_unk_piece: Emits the unk literal string (Default = false)
++ num_samples: How many samples to return (Default = 1)
++- theta: inverse temperature for sampling
+++ alpha: inverse temperature for sampling
++ wor: whether to sample without replacement (Default = false)
++ include_best: whether to include the best tokenization, requires wor=True (Default = false)
++ """
++@@ -470,8 +603,8 @@ class SentencePieceProcessor(object):
++ emit_unk_piece = self._emit_unk_piece
++ if num_samples is None:
++ num_samples = 1
++- if theta is None:
++- theta = 1.
+++ if alpha is None:
+++ alpha = 1.
++ if wor is None:
++ wor = False
++ if include_best is None:
++@@ -486,10 +619,10 @@ class SentencePieceProcessor(object):
++
++ def _encode(text):
++ if out_type is int:
++- return self._SampleEncodeAndScoreAsIds(text, num_samples, theta, wor, include_best,
+++ return self._SampleEncodeAndScoreAsIds(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++ else:
++- return self._SampleEncodeAndScoreAsPieces(text, num_samples, theta, wor, include_best,
+++ return self._SampleEncodeAndScoreAsPieces(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++
++ if type(input) is list:
++@@ -502,7 +635,7 @@ class SentencePieceProcessor(object):
++ """Decode processed id or token sequences.
++
++ Args:
++- out_type: output type. str or 'proto' (Default = str)
+++ out_type: output type. str or 'serialized_proto' or 'immutable_proto' (Default = str)
++ num_threads: the number of threads used in the batch processin (Default = 1).
++ """
++
++@@ -533,7 +666,7 @@ class SentencePieceProcessor(object):
++ if type(input[0][0]) is str:
++ return self._DecodePiecesBatch(input, num_threads)
++
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto':
++ if type(input) is int:
++ return self._DecodeIdsAsSerializedProto([input])
++ if type(input) is str:
++@@ -552,6 +685,25 @@ class SentencePieceProcessor(object):
++ return self._DecodePiecesAsSerializedProtoBatch(input, num_threads)
++
++
+++ if out_type == 'immutable_proto':
+++ if type(input) is int:
+++ return self._DecodeIdsAsImmutableProto([input])
+++ if type(input) is str:
+++ return self._DecodePiecesAsImmutableProto([input])
+++
+++ if type(input) is list:
+++ if len(input) == 0 or type(input[0]) is int:
+++ return self._DecodeIdsAsImmutableProto(input)
+++ if type(input[0]) is str:
+++ return self._DecodePiecesAsImmutableProto(input)
+++
+++ if type(input[0]) is list:
+++ if len(input[0]) == 0 or type(input[0][0]) is int:
+++ return self._DecodeIdsAsImmutableProtoBatch(input, num_threads)
+++ if type(input[0][0]) is str:
+++ return self._DecodePiecesAsImmutableProtoBatch(input, num_threads)
+++
+++
++ raise RuntimeError('unknown output or input type')
++ return None
++
++@@ -564,24 +716,32 @@ class SentencePieceProcessor(object):
++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++
++- def DecodePiecesAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ def DecodePiecesAsSerializedProto(self, input, out_type='serialized_proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def DecodeIdsAsSerializedProto(self, input, out_type='serialized_proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def DecodePiecesAsImmutableProto(self, input, out_type='immutable_proto', **kwargs):
++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++
++- def DecodeIdsAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ def DecodeIdsAsImmutableProto(self, input, out_type='immutable_proto', **kwargs):
++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++
++- def CalculateEntropy(self, input, theta, num_threads=None):
+++ def CalculateEntropy(self, input, alpha, num_threads=None):
++ """Calculate sentence entropy"""
++ if type(input) is list:
++ if num_threads is None:
++ num_threads = self._num_threads
++ if num_threads is None or type(num_threads) is not int:
++ raise RuntimeError('num_threads must be int')
++- return self._CalculateEntropyBatch(input, theta, num_threads)
+++ return self._CalculateEntropyBatch(input, alpha, num_threads)
++
++- return self._CalculateEntropy(input, theta)
+++ return self._CalculateEntropy(input, alpha)
++
++
++ def piece_size(self):
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 40373ce..1e2e1e0 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -166,7 +166,17 @@ inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
++ if (add_bos || add_eos || reverse || emit_unk_piece) {
++ throw sentencepiece::util::Status(
++ sentencepiece::util::StatusCode::kUnimplemented,
++- "add_bos, add_eos, reverse, and emit_unk_piece is not supported in AsSerialize API");
+++ "add_bos, add_eos, reverse, and emit_unk_piece is not supported in proto API");
+++ }
+++}
+++
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ sentencepiece::ImmutableSentencePieceText *proto,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
+++ if (add_bos || add_eos || reverse || emit_unk_piece) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kUnimplemented,
+++ "add_bos, add_eos, reverse, and emit_unk_piece is not supported in proto API");
++ }
++ }
++
++@@ -216,7 +226,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ #define DEFINE_ENCODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
++ std::vector<OutType> outs(ins.size()); \
++- InitNumThreads(ins, &num_threads); \
+++ InitNumThreads(ins, &num_threads); \
++ { \
++ ThreadPool pool(ins.size()); \
++ for (int n = 0; n < num_threads; ++n) { \
++@@ -237,7 +247,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ #define DEFINE_DECODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
++ std::vector<OutType> outs(ins.size()); \
++- InitNumThreads(ins, &num_threads); \
+++ InitNumThreads(ins, &num_threads); \
++ { \
++ ThreadPool pool(ins.size()); \
++ for (int n = 0; n < num_threads; ++n) { \
++@@ -264,6 +274,8 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++ }
++
+++%apply unsigned int { uint32_t }
+++
++ %ignore sentencepiece::util::Status;
++ %ignore sentencepiece::util::StatusCode;
++ %ignore absl::string_view;
++@@ -272,32 +284,48 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ %ignore sentencepiece::NormalizerSpec;
++ %ignore sentencepiece::TrainerSpec;
++ %ignore sentencepiece::SentencePieceProcessor::status;
+++%ignore sentencepiece::ImmutableSentencePieceText::mutable_proto;
+++%ignore sentencepiece::ImmutableSentencePieceText::pieces() const;
+++%ignore sentencepiece::ImmutableNBestSentencePieceText::mutable_proto;
+++%ignore sentencepiece::ImmutableNBestSentencePieceText::nbests() const;
++
++ %ignore sentencepiece::SentencePieceProcessor::Encode;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncode;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncode;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScore;
+++%ignore sentencepiece::SentencePieceProcessor::Decode;
+++
++ %ignore sentencepiece::SentencePieceProcessor::EncodeAsPieces;
++ %ignore sentencepiece::SentencePieceProcessor::EncodeAsIds;
++-%ignore sentencepiece::SentencePieceProcessor::EncodeAsSerializedProto;
++-%ignore sentencepiece::SentencePieceProcessor::SampleEncode;
++ %ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsIds;
++ %ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsPieces;
++-%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsSerializedProto;
++-%ignore sentencepiece::SentencePieceProcessor::NBestEncode;
++-%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsPieces;
++ %ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsIds;
++-%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsSerializedProto;
++-%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScore;
++-
++-%ignore sentencepiece::SentencePieceProcessor::Decode;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsPieces;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScoreAsIds;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScoreAsPieces;
++ %ignore sentencepiece::SentencePieceProcessor::DecodeIds;
++ %ignore sentencepiece::SentencePieceProcessor::DecodePieces;
+++
+++%ignore sentencepiece::SentencePieceProcessor::EncodeAsSerializedProto;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsSerializedProto;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsSerializedProto;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScoreAsSerializedProto;
++ %ignore sentencepiece::SentencePieceProcessor::DecodePiecesAsSerializedProto;
++ %ignore sentencepiece::SentencePieceProcessor::DecodeIdsAsSerializedProto;
++
+++%ignore sentencepiece::SentencePieceProcessor::EncodeAsImmutableProto;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAsImmutableProto;
+++%ignore sentencepiece::SentencePieceProcessor::NBestEncodeAsImmutableProto;
+++%ignore sentencepiece::SentencePieceProcessor::SampleEncodeAndScoreAsImmutableProto;
+++%ignore sentencepiece::SentencePieceProcessor::DecodePiecesAsImmutableProto;
+++%ignore sentencepiece::SentencePieceProcessor::DecodeIdsAsImmutableProto;
+++
++ %ignore sentencepiece::SentencePieceProcessor::model_proto;
++ %ignore sentencepiece::SentencePieceProcessor::Load;
++ %ignore sentencepiece::SentencePieceProcessor::LoadOrDie;
++ %ignore sentencepiece::pretokenizer::PretokenizerForTrainingInterface;
++ %ignore sentencepiece::SentenceIterator;
+++%ignore sentencepiece::ConvertToUnicodeSpans;
++ %ignore sentencepiece::SentencePieceTrainer::Train;
++ %ignore sentencepiece::SentencePieceTrainer::GetNormalizerSpec;
++ %ignore sentencepiece::SentencePieceTrainer::PopulateNormalizerSpec;
++@@ -351,6 +379,19 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return proto;
++ }
++
+++ sentencepiece::ImmutableSentencePieceText
+++ _EncodeAsImmutableProto(absl::string_view text,
+++ bool enable_sampling,
+++ int nbest_size, float alpha,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ auto proto = enable_sampling ?
+++ $self->SampleEncodeAsImmutableProto(text, nbest_size, alpha) :
+++ $self->EncodeAsImmutableProto(text);
+++ RewriteIds(*$self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
+++ return proto;
+++ }
+++
++ /////////////////////////////////////////////////////////////////////////////
++ // EncodeAs* (Batch request)
++ std::vector<std::vector<int>> _EncodeAsIdsBatch(
++@@ -381,6 +422,17 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ sentencepiece::util::bytes);
++ }
++
+++ std::vector<sentencepiece::ImmutableSentencePieceText>
+++ _EncodeAsImmutableProtoBatch(
+++ const std::vector<absl::string_view> &ins, int num_threads,
+++ bool enable_sampling, int nbest_size, float alpha,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsImmutableProto,
+++ absl::string_view,
+++ sentencepiece::ImmutableSentencePieceText);
+++ }
+++
++ /////////////////////////////////////////////////////////////////////////////
++ // DecodeAs* (Single request)
++ std::string _DecodeIds(const std::vector<int> &ids) const {
++@@ -404,6 +456,18 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return $self->DecodePiecesAsSerializedProto(pieces);
++ }
++
+++ sentencepiece::ImmutableSentencePieceText _DecodeIdsAsImmutableProto(
+++ const std::vector<int> &ids) const {
+++ CheckIds(ids, $self->GetPieceSize());
+++ return $self->DecodeIdsAsImmutableProto(ids);
+++ }
+++
+++ sentencepiece::ImmutableSentencePieceText _DecodePiecesAsImmutableProto(
+++ const std::vector<absl::string_view> &pieces) const {
+++ CheckIds(pieces, $self->GetPieceSize());
+++ return $self->DecodePiecesAsImmutableProto(pieces);
+++ }
+++
++ /////////////////////////////////////////////////////////////////////////////
++ // DecodeAs* (Batch request)
++ std::vector<std::string> _DecodeIdsBatch(
++@@ -428,6 +492,13 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ sentencepiece::util::bytes);
++ }
++
+++ std::vector<sentencepiece::ImmutableSentencePieceText>
+++ _DecodePiecesAsImmutableProtoBatch(
+++ const std::vector<std::vector<absl::string_view>> &ins, int num_threads) const {
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsImmutableProto, std::string,
+++ sentencepiece::ImmutableSentencePieceText);
+++ }
+++
++ ////////////////////////////////////////////////////////////////////////////
++ // NBestEncodeAs* (Single request)
++ std::vector<std::vector<int>>
++@@ -454,25 +525,37 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return piecess;
++ }
++
++- sentencepiece::util::bytes _NBestEncodeAsSerializedProto(absl::string_view text,
++- int nbest_size,
++- bool add_bos, bool add_eos, bool reverse,
++- bool emit_unk_piece) const {
+++ sentencepiece::util::bytes
+++ _NBestEncodeAsSerializedProto(absl::string_view text,
+++ int nbest_size,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
++ RewriteIds(*$self, static_cast<sentencepiece::util::bytes *>(nullptr),
++ add_bos, add_eos, reverse, emit_unk_piece);
++ return $self->NBestEncodeAsSerializedProto(text, nbest_size);
++ }
++
+++ sentencepiece::ImmutableNBestSentencePieceText
+++ _NBestEncodeAsImmutableProto(absl::string_view text,
+++ int nbest_size,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ RewriteIds(*$self, static_cast<sentencepiece::ImmutableSentencePieceText *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return $self->NBestEncodeAsImmutableProto(text, nbest_size);
+++ }
+++
+++
++ /////////////////////////////////////////////////////////////////////////////
++ // SampleEncodeAndScoreAs* (Single request)
++ std::vector<std::pair<std::vector<int>, float>>
++ _SampleEncodeAndScoreAsIds(absl::string_view text,
++- int num_samples, float theta, bool wor,
+++ int num_samples, float alpha, bool wor,
++ bool include_best,
++ bool add_bos, bool add_eos, bool reverse,
++ bool emit_unk_piece) const {
++ auto idss = $self->SampleEncodeAndScoreAsIds(text, num_samples,
++- theta, wor, include_best);
+++ alpha, wor, include_best);
++ for (auto &ids : idss) {
++ RewriteIds(*$self, &ids.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++@@ -481,25 +564,50 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ std::vector<std::pair<std::vector<std::string>, float>>
++ _SampleEncodeAndScoreAsPieces(absl::string_view text,
++- int num_samples, float theta, bool wor,
+++ int num_samples, float alpha, bool wor,
++ bool include_best,
++ bool add_bos, bool add_eos, bool reverse,
++ bool emit_unk_piece) const {
++ auto piecess = $self->SampleEncodeAndScoreAsPieces(text, num_samples,
++- theta, wor, include_best);
+++ alpha, wor, include_best);
++ for (auto &pieces : piecess) {
++ RewriteIds(*$self, &pieces.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return piecess;
++ }
++
+++ sentencepiece::util::bytes
+++ _SampleEncodeAndScoreAsSerializedProto(absl::string_view text,
+++ int num_samples, float alpha, bool wor,
+++ bool include_best,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ RewriteIds(*$self, static_cast<sentencepiece::util::bytes *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return $self->SampleEncodeAndScoreAsSerializedProto(text, num_samples,
+++ alpha, wor, include_best);
+++ }
+++
+++ sentencepiece::ImmutableNBestSentencePieceText
+++ _SampleEncodeAndScoreAsImmutableProto(absl::string_view text,
+++ int num_samples, float alpha, bool wor,
+++ bool include_best,
+++ bool add_bos, bool add_eos, bool reverse,
+++ bool emit_unk_piece) const {
+++ RewriteIds(*$self, static_cast<sentencepiece::util::bytes *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return $self->SampleEncodeAndScoreAsImmutableProto(text, num_samples,
+++ alpha, wor, include_best);
+++ }
+++
+++
++ // Calculate Entropy
++- float _CalculateEntropy(absl::string_view text, float theta) {
++- return $self->CalculateEntropy(text, theta);
+++ float _CalculateEntropy(absl::string_view text, float alpha) {
+++ return $self->CalculateEntropy(text, alpha);
++ }
++
++ std::vector<float> _CalculateEntropyBatch(const std::vector<absl::string_view> &ins,
++- float theta, int num_threads) {
+++ float alpha, int num_threads) {
++ std::vector<float> outs(ins.size());
++ InitNumThreads(ins, &num_threads);
++ {
++@@ -507,7 +615,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ for (int n = 0; n < num_threads; ++n) {
++ pool.Schedule([&, n]() {
++ for (size_t i = n; i < ins.size(); i += num_threads) {
++- outs[i] = self->CalculateEntropy(ins[i], theta);
+++ outs[i] = self->CalculateEntropy(ins[i], alpha);
++ }
++ });
++ }
++@@ -634,9 +742,12 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ if out_type is str:
++ return self._EncodeAsPiecesBatch(input, num_threads, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto' or out_type == 'proto':
++ return self._EncodeAsSerializedProtoBatch(input, num_threads, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'immutable_proto':
+++ return self._EncodeAsImmutableProtoBatch(input, num_threads, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
++ if out_type is int:
++ return self._EncodeAsIds(input, enable_sampling, nbest_size,
++@@ -644,9 +755,12 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ if out_type is str:
++ return self._EncodeAsPieces(input, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto' or out_type == 'proto':
++ return self._EncodeAsSerializedProto(input, enable_sampling, nbest_size,
++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'immutable_proto':
+++ return self._EncodeAsImmutableProto(input, enable_sampling, nbest_size,
+++ alpha, add_bos, add_eos, reverse, emit_unk_piece)
++
++ raise RuntimeError('unknown out_type={}'.format(out_type))
++ return None
++@@ -661,7 +775,11 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++
++ def EncodeAsSerializedProto(self, input, **kwargs):
++- return self.Encode(input=input, out_type='proto', **kwargs)
+++ return self.Encode(input=input, out_type='serialized_proto', **kwargs)
+++
+++
+++ def EncodeAsImmutableProto(self, input, **kwargs):
+++ return self.Encode(input=input, out_type='immutable_proto', **kwargs)
++
++
++ def SampleEncodeAsPieces(self, input, nbest_size=None, alpha=None, **kwargs):
++@@ -676,7 +794,12 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ def SampleEncodeAsSerializedProto(self, input, nbest_size=None, alpha=None, **kwargs):
++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
++- out_type='proto', enable_sampling=True, **kwargs)
+++ out_type='serialized_proto', enable_sampling=True, **kwargs)
+++
+++
+++ def SampleEncodeAsImmutableProto(self, input, nbest_size=None, alpha=None, **kwargs):
+++ return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
+++ out_type='immutable_proto', enable_sampling=True, **kwargs)
++
++
++ def NBestEncode(self,
++@@ -722,9 +845,12 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ if out_type is str:
++ return self._NBestEncodeAsPieces(text, nbest_size,
++ add_bos, add_eos, reverse, emit_unk_piece)
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto' or out_type == 'proto':
++ return self._NBestEncodeAsSerializedProto(text, nbest_size,
++ add_bos, add_eos, reverse, emit_unk_piece)
+++ if out_type == 'immutable_proto':
+++ return self._NBestEncodeAsImmutableProto(text, nbest_size,
+++ add_bos, add_eos, reverse, emit_unk_piece)
++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++@@ -744,7 +870,12 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ def NBestEncodeAsSerializedProto(self, input, nbest_size=None, **kwargs):
++ return self.NBestEncode(input=input, nbest_size=nbest_size,
++- out_type='proto', **kwargs)
+++ out_type='serialized_proto', **kwargs)
+++
+++
+++ def NBestEncodeAsImmutableProto(self, input, nbest_size=None, **kwargs):
+++ return self.NBestEncode(input=input, nbest_size=nbest_size,
+++ out_type='immutable_proto', **kwargs)
++
++
++ def SampleEncodeAndScore(self,
++@@ -755,20 +886,20 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ reverse=None,
++ emit_unk_piece=None,
++ num_samples=None,
++- theta=None,
+++ alpha=None,
++ wor=None,
++ include_best=None):
++ """SampleEncodeAndScore text input to segmented ids or tokens.
++
++ Args:
++ input: input string. accepsts list of string.
++- out_type: output type. int or str or 'proto'.
+++ out_type: output type. int or str or 'serialized_proto' or 'immutable_proto'
++ add_bos: Add <s> to the result (Default = false)
++ add_eos: Add </s> to the result (Default = false) <s>/</s> is added after reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++ emit_unk_piece: Emits the unk literal string (Default = false)
++ num_samples: How many samples to return (Default = 1)
++- theta: inverse temperature for sampling
+++ alpha: inverse temperature for sampling
++ wor: whether to sample without replacement (Default = false)
++ include_best: whether to include the best tokenization, requires wor=True (Default = false)
++ """
++@@ -785,8 +916,8 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ emit_unk_piece = self._emit_unk_piece
++ if num_samples is None:
++ num_samples = 1
++- if theta is None:
++- theta = 1.
+++ if alpha is None:
+++ alpha = 1.
++ if wor is None:
++ wor = False
++ if include_best is None:
++@@ -801,10 +932,10 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ def _encode(text):
++ if out_type is int:
++- return self._SampleEncodeAndScoreAsIds(text, num_samples, theta, wor, include_best,
+++ return self._SampleEncodeAndScoreAsIds(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++ else:
++- return self._SampleEncodeAndScoreAsPieces(text, num_samples, theta, wor, include_best,
+++ return self._SampleEncodeAndScoreAsPieces(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++
++ if type(input) is list:
++@@ -817,7 +948,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ """Decode processed id or token sequences.
++
++ Args:
++- out_type: output type. str or 'proto' (Default = str)
+++ out_type: output type. str or 'serialized_proto' or 'immutable_proto' (Default = str)
++ num_threads: the number of threads used in the batch processin (Default = 1).
++ """
++
++@@ -848,7 +979,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ if type(input[0][0]) is str:
++ return self._DecodePiecesBatch(input, num_threads)
++
++- if out_type == 'proto':
+++ if out_type == 'serialized_proto':
++ if type(input) is int:
++ return self._DecodeIdsAsSerializedProto([input])
++ if type(input) is str:
++@@ -867,6 +998,25 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return self._DecodePiecesAsSerializedProtoBatch(input, num_threads)
++
++
+++ if out_type == 'immutable_proto':
+++ if type(input) is int:
+++ return self._DecodeIdsAsImmutableProto([input])
+++ if type(input) is str:
+++ return self._DecodePiecesAsImmutableProto([input])
+++
+++ if type(input) is list:
+++ if len(input) == 0 or type(input[0]) is int:
+++ return self._DecodeIdsAsImmutableProto(input)
+++ if type(input[0]) is str:
+++ return self._DecodePiecesAsImmutableProto(input)
+++
+++ if type(input[0]) is list:
+++ if len(input[0]) == 0 or type(input[0][0]) is int:
+++ return self._DecodeIdsAsImmutableProtoBatch(input, num_threads)
+++ if type(input[0][0]) is str:
+++ return self._DecodePiecesAsImmutableProtoBatch(input, num_threads)
+++
+++
++ raise RuntimeError('unknown output or input type')
++ return None
++
++@@ -879,24 +1029,32 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++
++- def DecodePiecesAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ def DecodePiecesAsSerializedProto(self, input, out_type='serialized_proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def DecodeIdsAsSerializedProto(self, input, out_type='serialized_proto', **kwargs):
+++ return self.Decode(input=input, out_type=out_type, **kwargs)
+++
+++
+++ def DecodePiecesAsImmutableProto(self, input, out_type='immutable_proto', **kwargs):
++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++
++- def DecodeIdsAsSerializedProto(self, input, out_type='proto', **kwargs):
+++ def DecodeIdsAsImmutableProto(self, input, out_type='immutable_proto', **kwargs):
++ return self.Decode(input=input, out_type=out_type, **kwargs)
++
++
++- def CalculateEntropy(self, input, theta, num_threads=None):
+++ def CalculateEntropy(self, input, alpha, num_threads=None):
++ """Calculate sentence entropy"""
++ if type(input) is list:
++ if num_threads is None:
++ num_threads = self._num_threads
++ if num_threads is None or type(num_threads) is not int:
++ raise RuntimeError('num_threads must be int')
++- return self._CalculateEntropyBatch(input, theta, num_threads)
+++ return self._CalculateEntropyBatch(input, alpha, num_threads)
++
++- return self._CalculateEntropy(input, theta)
+++ return self._CalculateEntropy(input, alpha)
++
++
++ def piece_size(self):
++@@ -1028,6 +1186,50 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++ }
++
+++%extend sentencepiece::ImmutableSentencePieceText {
+++ ImmutableSentencePieceText_ImmutableSentencePiece pieces(int index) const {
+++ if (index < 0 || index >= static_cast<int>($self->pieces_size())) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kOutOfRange,
+++ "piece index is out of range.");
+++ }
+++ return $self->pieces(index);
+++ }
+++
+++%pythoncode {
+++ def __len__(self):
+++ return self.pieces_size()
+++
+++ def __getitem__(self, i):
+++ return self.pieces(i)
+++
+++ def __eq__(self, other):
+++ return self.SerializeAsString() == other.SerializeAsString()
+++}
+++}
+++
+++%extend sentencepiece::ImmutableNBestSentencePieceText {
+++ ImmutableSentencePieceText nbests(int index) const {
+++ if (index < 0 || index >= static_cast<int>($self->nbests_size())) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kOutOfRange,
+++ "nbest index is out of range.");
+++ }
+++ return $self->nbests(index);
+++ }
+++
+++%pythoncode {
+++ def __len__(self):
+++ return self.nbests_size()
+++
+++ def __getitem__(self, i):
+++ return self.nbests(i)
+++
+++ def __eq__(self, other):
+++ return self.SerializeAsString() == other.SerializeAsString()
+++}
+++}
+++
++ %typemap(out) std::vector<int> {
++ $result = PyList_New($1.size());
++ for (size_t i = 0; i < $1.size(); ++i) {
++@@ -1277,6 +1479,14 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++ }
++
+++%typemap(out) std::vector<sentencepiece::ImmutableSentencePieceText> {
+++ $result = PyList_New($1.size());
+++ for (size_t i = 0; i < $1.size(); ++i) {
+++ PyObject *obj = SWIG_NewPointerObj(new sentencepiece::ImmutableSentencePieceText($1.at(i)), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0);
+++ PyList_SET_ITEM($result, i, obj);
+++ }
+++}
+++
++ %typemap(in) sentencepiece::SentenceIterator * {
++ sentencepiece::SentenceIterator *out = nullptr;
++ if (PyIter_Check($input)) {
++@@ -1324,6 +1534,18 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ delete $1;
++ }
++
+++%typemap(freearg) sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece {
+++ delete $1;
+++}
+++
+++%typemap(freearg) sentencepiece::ImmutableSentencePieceText {
+++ delete $1;
+++}
+++
+++%typemap(freearg) sentencepiece::ImmutableNBestSentencePieceText {
+++ delete $1;
+++}
+++
++ %include <sentencepiece_processor.h>
++ %include <sentencepiece_trainer.h>
++
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 36ce38c..9776b0f 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -2694,17 +2694,20 @@ SWIGINTERN PyObject *SWIG_PyStaticMethod_New(PyObject *SWIGUNUSEDPARM(self), PyO
++
++ #define SWIGTYPE_p_char swig_types[0]
++ #define SWIGTYPE_p_float swig_types[1]
++-#define SWIGTYPE_p_sentencepiece__SentenceIterator swig_types[2]
++-#define SWIGTYPE_p_sentencepiece__SentencePieceProcessor swig_types[3]
++-#define SWIGTYPE_p_sentencepiece__SentencePieceTrainer swig_types[4]
++-#define SWIGTYPE_p_std__string swig_types[5]
++-#define SWIGTYPE_p_std__unordered_mapT_std__string_std__string_t swig_types[6]
++-#define SWIGTYPE_p_std__vectorT_absl__string_view_t swig_types[7]
++-#define SWIGTYPE_p_std__vectorT_int_t swig_types[8]
++-#define SWIGTYPE_p_std__vectorT_std__vectorT_absl__string_view_t_t swig_types[9]
++-#define SWIGTYPE_p_std__vectorT_std__vectorT_int_t_t swig_types[10]
++-static swig_type_info *swig_types[12];
++-static swig_module_info swig_module = {swig_types, 11, 0, 0, 0, 0};
+++#define SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText swig_types[2]
+++#define SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText swig_types[3]
+++#define SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece swig_types[4]
+++#define SWIGTYPE_p_sentencepiece__SentenceIterator swig_types[5]
+++#define SWIGTYPE_p_sentencepiece__SentencePieceProcessor swig_types[6]
+++#define SWIGTYPE_p_sentencepiece__SentencePieceTrainer swig_types[7]
+++#define SWIGTYPE_p_std__string swig_types[8]
+++#define SWIGTYPE_p_std__unordered_mapT_std__string_std__string_t swig_types[9]
+++#define SWIGTYPE_p_std__vectorT_absl__string_view_t swig_types[10]
+++#define SWIGTYPE_p_std__vectorT_int_t swig_types[11]
+++#define SWIGTYPE_p_std__vectorT_std__vectorT_absl__string_view_t_t swig_types[12]
+++#define SWIGTYPE_p_std__vectorT_std__vectorT_int_t_t swig_types[13]
+++static swig_type_info *swig_types[15];
+++static swig_module_info swig_module = {swig_types, 14, 0, 0, 0, 0};
++ #define SWIG_TypeQuery(name) SWIG_TypeQueryModule(&swig_module, &swig_module, name)
++ #define SWIG_MangledTypeQuery(name) SWIG_MangledTypeQueryModule(&swig_module, &swig_module, name)
++
++@@ -2972,7 +2975,17 @@ inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
++ if (add_bos || add_eos || reverse || emit_unk_piece) {
++ throw sentencepiece::util::Status(
++ sentencepiece::util::StatusCode::kUnimplemented,
++- "add_bos, add_eos, reverse, and emit_unk_piece is not supported in AsSerialize API");
+++ "add_bos, add_eos, reverse, and emit_unk_piece is not supported in proto API");
+++ }
+++}
+++
+++inline void RewriteIds(const sentencepiece::SentencePieceProcessor &sp,
+++ sentencepiece::ImmutableSentencePieceText *proto,
+++ bool add_bos, bool add_eos, bool reverse, bool emit_unk_piece) {
+++ if (add_bos || add_eos || reverse || emit_unk_piece) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kUnimplemented,
+++ "add_bos, add_eos, reverse, and emit_unk_piece is not supported in proto API");
++ }
++ }
++
++@@ -3022,7 +3035,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ #define DEFINE_ENCODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
++ std::vector<OutType> outs(ins.size()); \
++- InitNumThreads(ins, &num_threads); \
+++ InitNumThreads(ins, &num_threads); \
++ { \
++ ThreadPool pool(ins.size()); \
++ for (int n = 0; n < num_threads; ++n) { \
++@@ -3043,7 +3056,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ #define DEFINE_DECODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
++ std::vector<OutType> outs(ins.size()); \
++- InitNumThreads(ins, &num_threads); \
+++ InitNumThreads(ins, &num_threads); \
++ { \
++ ThreadPool pool(ins.size()); \
++ for (int n = 0; n < num_threads; ++n) { \
++@@ -3060,131 +3073,24 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ } // namespace
++
++
++-SWIGINTERN swig_type_info*
++-SWIG_pchar_descriptor(void)
+++SWIGINTERNINLINE PyObject*
+++ SWIG_From_unsigned_SS_int (unsigned int value)
++ {
++- static int init = 0;
++- static swig_type_info* info = 0;
++- if (!init) {
++- info = SWIG_TypeQuery("_p_char");
++- init = 1;
++- }
++- return info;
+++ return PyInt_FromSize_t((size_t) value);
++ }
++
++
++-SWIGINTERN int
++-SWIG_AsCharPtrAndSize(PyObject *obj, char** cptr, size_t* psize, int *alloc)
++-{
++-#if PY_VERSION_HEX>=0x03000000
++-#if defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
++- if (PyBytes_Check(obj))
++-#else
++- if (PyUnicode_Check(obj))
++-#endif
++-#else
++- if (PyString_Check(obj))
++-#endif
++- {
++- char *cstr; Py_ssize_t len;
++- int ret = SWIG_OK;
++-#if PY_VERSION_HEX>=0x03000000
++-#if !defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
++- if (!alloc && cptr) {
++- /* We can't allow converting without allocation, since the internal
++- representation of string in Python 3 is UCS-2/UCS-4 but we require
++- a UTF-8 representation.
++- TODO(bhy) More detailed explanation */
++- return SWIG_RuntimeError;
++- }
++- obj = PyUnicode_AsUTF8String(obj);
++- if (!obj)
++- return SWIG_TypeError;
++- if (alloc)
++- *alloc = SWIG_NEWOBJ;
++-#endif
++- if (PyBytes_AsStringAndSize(obj, &cstr, &len) == -1)
++- return SWIG_TypeError;
++-#else
++- if (PyString_AsStringAndSize(obj, &cstr, &len) == -1)
++- return SWIG_TypeError;
++-#endif
++- if (cptr) {
++- if (alloc) {
++- if (*alloc == SWIG_NEWOBJ) {
++- *cptr = reinterpret_cast< char* >(memcpy(new char[len + 1], cstr, sizeof(char)*(len + 1)));
++- *alloc = SWIG_NEWOBJ;
++- } else {
++- *cptr = cstr;
++- *alloc = SWIG_OLDOBJ;
++- }
++- } else {
++-#if PY_VERSION_HEX>=0x03000000
++-#if defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
++- *cptr = PyBytes_AsString(obj);
++-#else
++- assert(0); /* Should never reach here with Unicode strings in Python 3 */
++-#endif
++-#else
++- *cptr = SWIG_Python_str_AsChar(obj);
++- if (!*cptr)
++- ret = SWIG_TypeError;
++-#endif
++- }
++- }
++- if (psize) *psize = len + 1;
++-#if PY_VERSION_HEX>=0x03000000 && !defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
++- Py_XDECREF(obj);
++-#endif
++- return ret;
++- } else {
++-#if defined(SWIG_PYTHON_2_UNICODE)
++-#if defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
++-#error "Cannot use both SWIG_PYTHON_2_UNICODE and SWIG_PYTHON_STRICT_BYTE_CHAR at once"
++-#endif
++-#if PY_VERSION_HEX<0x03000000
++- if (PyUnicode_Check(obj)) {
++- char *cstr; Py_ssize_t len;
++- if (!alloc && cptr) {
++- return SWIG_RuntimeError;
++- }
++- obj = PyUnicode_AsUTF8String(obj);
++- if (!obj)
++- return SWIG_TypeError;
++- if (PyString_AsStringAndSize(obj, &cstr, &len) != -1) {
++- if (cptr) {
++- if (alloc) *alloc = SWIG_NEWOBJ;
++- *cptr = reinterpret_cast< char* >(memcpy(new char[len + 1], cstr, sizeof(char)*(len + 1)));
++- }
++- if (psize) *psize = len + 1;
+++ #define SWIG_From_long PyInt_FromLong
++
++- Py_XDECREF(obj);
++- return SWIG_OK;
++- } else {
++- Py_XDECREF(obj);
++- }
++- }
++-#endif
++-#endif
++
++- swig_type_info* pchar_descriptor = SWIG_pchar_descriptor();
++- if (pchar_descriptor) {
++- void* vptr = 0;
++- if (SWIG_ConvertPtr(obj, &vptr, pchar_descriptor, 0) == SWIG_OK) {
++- if (cptr) *cptr = (char *) vptr;
++- if (psize) *psize = vptr ? (strlen((char *)vptr) + 1) : 0;
++- if (alloc) *alloc = SWIG_OLDOBJ;
++- return SWIG_OK;
++- }
++- }
++- }
++- return SWIG_TypeError;
+++SWIGINTERNINLINE PyObject*
+++SWIG_From_unsigned_SS_long (unsigned long value)
+++{
+++ return (value > LONG_MAX) ?
+++ PyLong_FromUnsignedLong(value) : PyInt_FromLong(static_cast< long >(value));
++ }
++
++
++-
++-
++-
++ #include <limits.h>
++ #if !defined(SWIG_NO_LLONG_MAX)
++ # if !defined(LLONG_MAX) && defined(__GNUC__) && defined (__LONG_LONG_MAX__)
++@@ -3195,6 +3101,47 @@ SWIG_AsCharPtrAndSize(PyObject *obj, char** cptr, size_t* psize, int *alloc)
++ #endif
++
++
+++#if defined(LLONG_MAX) && !defined(SWIG_LONG_LONG_AVAILABLE)
+++# define SWIG_LONG_LONG_AVAILABLE
+++#endif
+++
+++
+++#ifdef SWIG_LONG_LONG_AVAILABLE
+++SWIGINTERNINLINE PyObject*
+++SWIG_From_unsigned_SS_long_SS_long (unsigned long long value)
+++{
+++ return (value > LONG_MAX) ?
+++ PyLong_FromUnsignedLongLong(value) : PyInt_FromLong(static_cast< long >(value));
+++}
+++#endif
+++
+++
+++SWIGINTERNINLINE PyObject *
+++SWIG_From_size_t (size_t value)
+++{
+++#ifdef SWIG_LONG_LONG_AVAILABLE
+++ if (sizeof(size_t) <= sizeof(unsigned long)) {
+++#endif
+++ return SWIG_From_unsigned_SS_long (static_cast< unsigned long >(value));
+++#ifdef SWIG_LONG_LONG_AVAILABLE
+++ } else {
+++ /* assume sizeof(size_t) <= sizeof(unsigned long long) */
+++ return SWIG_From_unsigned_SS_long_SS_long (static_cast< unsigned long long >(value));
+++ }
+++#endif
+++}
+++
+++
+++ #define SWIG_From_double PyFloat_FromDouble
+++
+++
+++SWIGINTERNINLINE PyObject *
+++SWIG_From_float (float value)
+++{
+++ return SWIG_From_double (value);
+++}
+++
+++
++ SWIGINTERN int
++ SWIG_AsVal_double (PyObject *obj, double *val)
++ {
++@@ -3335,98 +3282,215 @@ SWIG_AsVal_int (PyObject * obj, int *val)
++ return res;
++ }
++
++-
++-/* Getting isfinite working pre C99 across multiple platforms is non-trivial. Users can provide SWIG_isfinite on older platforms. */
++-#ifndef SWIG_isfinite
++-/* isfinite() is a macro for C99 */
++-# if defined(isfinite)
++-# define SWIG_isfinite(X) (isfinite(X))
++-# elif defined(__cplusplus) && __cplusplus >= 201103L
++-/* Use a template so that this works whether isfinite() is std::isfinite() or
++- * in the global namespace. The reality seems to vary between compiler
++- * versions.
++- *
++- * Make sure namespace std exists to avoid compiler warnings.
++- *
++- * extern "C++" is required as this fragment can end up inside an extern "C" { } block
++- */
++-namespace std { }
++-extern "C++" template<typename T>
++-inline int SWIG_isfinite_func(T x) {
++- using namespace std;
++- return isfinite(x);
++-}
++-# define SWIG_isfinite(X) (SWIG_isfinite_func(X))
++-# elif defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 2))
++-# define SWIG_isfinite(X) (__builtin_isfinite(X))
++-# elif defined(__clang__) && defined(__has_builtin)
++-# if __has_builtin(__builtin_isfinite)
++-# define SWIG_isfinite(X) (__builtin_isfinite(X))
++-# endif
++-# elif defined(_MSC_VER)
++-# define SWIG_isfinite(X) (_finite(X))
++-# elif defined(__sun) && defined(__SVR4)
++-# include <ieeefp.h>
++-# define SWIG_isfinite(X) (finite(X))
++-# endif
++-#endif
++-
++-
++-/* Accept infinite as a valid float value unless we are unable to check if a value is finite */
++-#ifdef SWIG_isfinite
++-# define SWIG_Float_Overflow_Check(X) ((X < -FLT_MAX || X > FLT_MAX) && SWIG_isfinite(X))
++-#else
++-# define SWIG_Float_Overflow_Check(X) ((X < -FLT_MAX || X > FLT_MAX))
++-#endif
++-
++-
++-SWIGINTERN int
++-SWIG_AsVal_float (PyObject * obj, float *val)
++-{
++- double v;
++- int res = SWIG_AsVal_double (obj, &v);
++- if (SWIG_IsOK(res)) {
++- if (SWIG_Float_Overflow_Check(v)) {
++- return SWIG_OverflowError;
++- } else {
++- if (val) *val = static_cast< float >(v);
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece sentencepiece_ImmutableSentencePieceText_pieces(sentencepiece::ImmutableSentencePieceText const *self,int index){
+++ if (index < 0 || index >= static_cast<int>(self->pieces_size())) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kOutOfRange,
+++ "piece index is out of range.");
++ }
++- }
++- return res;
++-}
++-
+++ return self->pieces(index);
+++ }
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_ImmutableNBestSentencePieceText_nbests(sentencepiece::ImmutableNBestSentencePieceText const *self,int index){
+++ if (index < 0 || index >= static_cast<int>(self->nbests_size())) {
+++ throw sentencepiece::util::Status(
+++ sentencepiece::util::StatusCode::kOutOfRange,
+++ "nbest index is out of range.");
+++ }
+++ return self->nbests(index);
+++ }
++
++-SWIGINTERN int
++-SWIG_AsVal_bool (PyObject *obj, bool *val)
+++SWIGINTERN swig_type_info*
+++SWIG_pchar_descriptor(void)
++ {
++- int r;
++- if (!PyBool_Check(obj))
++- return SWIG_ERROR;
++- r = PyObject_IsTrue(obj);
++- if (r == -1)
++- return SWIG_ERROR;
++- if (val) *val = r ? true : false;
++- return SWIG_OK;
++-}
++-
++-
++- #define SWIG_From_double PyFloat_FromDouble
++-
++-
++-SWIGINTERNINLINE PyObject *
++-SWIG_From_float (float value)
++-{
++- return SWIG_From_double (value);
+++ static int init = 0;
+++ static swig_type_info* info = 0;
+++ if (!init) {
+++ info = SWIG_TypeQuery("_p_char");
+++ init = 1;
+++ }
+++ return info;
++ }
++
++
++-SWIGINTERNINLINE PyObject*
++- SWIG_From_int (int value)
+++SWIGINTERN int
+++SWIG_AsCharPtrAndSize(PyObject *obj, char** cptr, size_t* psize, int *alloc)
++ {
++- return PyInt_FromLong((long) value);
++-}
++-
++-
+++#if PY_VERSION_HEX>=0x03000000
+++#if defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
+++ if (PyBytes_Check(obj))
+++#else
+++ if (PyUnicode_Check(obj))
+++#endif
+++#else
+++ if (PyString_Check(obj))
+++#endif
+++ {
+++ char *cstr; Py_ssize_t len;
+++ int ret = SWIG_OK;
+++#if PY_VERSION_HEX>=0x03000000
+++#if !defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
+++ if (!alloc && cptr) {
+++ /* We can't allow converting without allocation, since the internal
+++ representation of string in Python 3 is UCS-2/UCS-4 but we require
+++ a UTF-8 representation.
+++ TODO(bhy) More detailed explanation */
+++ return SWIG_RuntimeError;
+++ }
+++ obj = PyUnicode_AsUTF8String(obj);
+++ if (!obj)
+++ return SWIG_TypeError;
+++ if (alloc)
+++ *alloc = SWIG_NEWOBJ;
+++#endif
+++ if (PyBytes_AsStringAndSize(obj, &cstr, &len) == -1)
+++ return SWIG_TypeError;
+++#else
+++ if (PyString_AsStringAndSize(obj, &cstr, &len) == -1)
+++ return SWIG_TypeError;
+++#endif
+++ if (cptr) {
+++ if (alloc) {
+++ if (*alloc == SWIG_NEWOBJ) {
+++ *cptr = reinterpret_cast< char* >(memcpy(new char[len + 1], cstr, sizeof(char)*(len + 1)));
+++ *alloc = SWIG_NEWOBJ;
+++ } else {
+++ *cptr = cstr;
+++ *alloc = SWIG_OLDOBJ;
+++ }
+++ } else {
+++#if PY_VERSION_HEX>=0x03000000
+++#if defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
+++ *cptr = PyBytes_AsString(obj);
+++#else
+++ assert(0); /* Should never reach here with Unicode strings in Python 3 */
+++#endif
+++#else
+++ *cptr = SWIG_Python_str_AsChar(obj);
+++ if (!*cptr)
+++ ret = SWIG_TypeError;
+++#endif
+++ }
+++ }
+++ if (psize) *psize = len + 1;
+++#if PY_VERSION_HEX>=0x03000000 && !defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
+++ Py_XDECREF(obj);
+++#endif
+++ return ret;
+++ } else {
+++#if defined(SWIG_PYTHON_2_UNICODE)
+++#if defined(SWIG_PYTHON_STRICT_BYTE_CHAR)
+++#error "Cannot use both SWIG_PYTHON_2_UNICODE and SWIG_PYTHON_STRICT_BYTE_CHAR at once"
+++#endif
+++#if PY_VERSION_HEX<0x03000000
+++ if (PyUnicode_Check(obj)) {
+++ char *cstr; Py_ssize_t len;
+++ if (!alloc && cptr) {
+++ return SWIG_RuntimeError;
+++ }
+++ obj = PyUnicode_AsUTF8String(obj);
+++ if (!obj)
+++ return SWIG_TypeError;
+++ if (PyString_AsStringAndSize(obj, &cstr, &len) != -1) {
+++ if (cptr) {
+++ if (alloc) *alloc = SWIG_NEWOBJ;
+++ *cptr = reinterpret_cast< char* >(memcpy(new char[len + 1], cstr, sizeof(char)*(len + 1)));
+++ }
+++ if (psize) *psize = len + 1;
+++
+++ Py_XDECREF(obj);
+++ return SWIG_OK;
+++ } else {
+++ Py_XDECREF(obj);
+++ }
+++ }
+++#endif
+++#endif
+++
+++ swig_type_info* pchar_descriptor = SWIG_pchar_descriptor();
+++ if (pchar_descriptor) {
+++ void* vptr = 0;
+++ if (SWIG_ConvertPtr(obj, &vptr, pchar_descriptor, 0) == SWIG_OK) {
+++ if (cptr) *cptr = (char *) vptr;
+++ if (psize) *psize = vptr ? (strlen((char *)vptr) + 1) : 0;
+++ if (alloc) *alloc = SWIG_OLDOBJ;
+++ return SWIG_OK;
+++ }
+++ }
+++ }
+++ return SWIG_TypeError;
+++}
+++
+++
+++
+++
+++
+++/* Getting isfinite working pre C99 across multiple platforms is non-trivial. Users can provide SWIG_isfinite on older platforms. */
+++#ifndef SWIG_isfinite
+++/* isfinite() is a macro for C99 */
+++# if defined(isfinite)
+++# define SWIG_isfinite(X) (isfinite(X))
+++# elif defined(__cplusplus) && __cplusplus >= 201103L
+++/* Use a template so that this works whether isfinite() is std::isfinite() or
+++ * in the global namespace. The reality seems to vary between compiler
+++ * versions.
+++ *
+++ * Make sure namespace std exists to avoid compiler warnings.
+++ *
+++ * extern "C++" is required as this fragment can end up inside an extern "C" { } block
+++ */
+++namespace std { }
+++extern "C++" template<typename T>
+++inline int SWIG_isfinite_func(T x) {
+++ using namespace std;
+++ return isfinite(x);
+++}
+++# define SWIG_isfinite(X) (SWIG_isfinite_func(X))
+++# elif defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 2))
+++# define SWIG_isfinite(X) (__builtin_isfinite(X))
+++# elif defined(__clang__) && defined(__has_builtin)
+++# if __has_builtin(__builtin_isfinite)
+++# define SWIG_isfinite(X) (__builtin_isfinite(X))
+++# endif
+++# elif defined(_MSC_VER)
+++# define SWIG_isfinite(X) (_finite(X))
+++# elif defined(__sun) && defined(__SVR4)
+++# include <ieeefp.h>
+++# define SWIG_isfinite(X) (finite(X))
+++# endif
+++#endif
+++
+++
+++/* Accept infinite as a valid float value unless we are unable to check if a value is finite */
+++#ifdef SWIG_isfinite
+++# define SWIG_Float_Overflow_Check(X) ((X < -FLT_MAX || X > FLT_MAX) && SWIG_isfinite(X))
+++#else
+++# define SWIG_Float_Overflow_Check(X) ((X < -FLT_MAX || X > FLT_MAX))
+++#endif
+++
+++
+++SWIGINTERN int
+++SWIG_AsVal_float (PyObject * obj, float *val)
+++{
+++ double v;
+++ int res = SWIG_AsVal_double (obj, &v);
+++ if (SWIG_IsOK(res)) {
+++ if (SWIG_Float_Overflow_Check(v)) {
+++ return SWIG_OverflowError;
+++ } else {
+++ if (val) *val = static_cast< float >(v);
+++ }
+++ }
+++ return res;
+++}
+++
+++
+++SWIGINTERNINLINE PyObject*
+++ SWIG_From_int (int value)
+++{
+++ return PyInt_FromLong((long) value);
+++}
+++
+++
++ SWIGINTERNINLINE PyObject*
++ SWIG_From_bool (bool value)
++ {
++@@ -3436,6 +3500,20 @@ SWIGINTERNINLINE PyObject*
++ SWIGINTERN sentencepiece::util::Status sentencepiece_SentencePieceProcessor_LoadFromFile(sentencepiece::SentencePieceProcessor *self,absl::string_view arg){
++ return self->Load(arg);
++ }
+++
+++SWIGINTERN int
+++SWIG_AsVal_bool (PyObject *obj, bool *val)
+++{
+++ int r;
+++ if (!PyBool_Check(obj))
+++ return SWIG_ERROR;
+++ r = PyObject_IsTrue(obj);
+++ if (r == -1)
+++ return SWIG_ERROR;
+++ if (val) *val = r ? true : false;
+++ return SWIG_OK;
+++}
+++
++ SWIGINTERN std::vector< int > sentencepiece_SentencePieceProcessor__EncodeAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto ids = enable_sampling ?
++ self->SampleEncodeAsIds(text, nbest_size, alpha) :
++@@ -3457,6 +3535,13 @@ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__Enco
++ RewriteIds(*self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
++ return proto;
++ }
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_SentencePieceProcessor__EncodeAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ auto proto = enable_sampling ?
+++ self->SampleEncodeAsImmutableProto(text, nbest_size, alpha) :
+++ self->EncodeAsImmutableProto(text);
+++ RewriteIds(*self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
+++ return proto;
+++ }
++ SWIGINTERN std::vector< std::vector< int > > sentencepiece_SentencePieceProcessor__EncodeAsIdsBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &ins,int num_threads,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsIds,
++ absl::string_view, std::vector<int>);
++@@ -3470,6 +3555,11 @@ SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__EncodeAsSerializedPr
++ absl::string_view,
++ sentencepiece::util::bytes);
++ }
+++SWIGINTERN std::vector< sentencepiece::ImmutableSentencePieceText > sentencepiece_SentencePieceProcessor__EncodeAsImmutableProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &ins,int num_threads,bool enable_sampling,int nbest_size,float alpha,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ DEFINE_ENCODE_BATCH_FUNC_IMPL(EncodeAsImmutableProto,
+++ absl::string_view,
+++ sentencepiece::ImmutableSentencePieceText);
+++ }
++ SWIGINTERN std::string sentencepiece_SentencePieceProcessor__DecodeIds(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
++ CheckIds(ids, self->GetPieceSize());
++ return self->DecodeIds(ids);
++@@ -3485,6 +3575,14 @@ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__Deco
++ CheckIds(pieces, self->GetPieceSize());
++ return self->DecodePiecesAsSerializedProto(pieces);
++ }
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_SentencePieceProcessor__DecodeIdsAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
+++ CheckIds(ids, self->GetPieceSize());
+++ return self->DecodeIdsAsImmutableProto(ids);
+++ }
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_SentencePieceProcessor__DecodePiecesAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &pieces){
+++ CheckIds(pieces, self->GetPieceSize());
+++ return self->DecodePiecesAsImmutableProto(pieces);
+++ }
++ SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodeIdsBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< int > > const &ins,int num_threads){
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIds, int, std::string);
++ }
++@@ -3499,6 +3597,10 @@ SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodePiecesAsSerial
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsSerializedProto, std::string,
++ sentencepiece::util::bytes);
++ }
+++SWIGINTERN std::vector< sentencepiece::ImmutableSentencePieceText > sentencepiece_SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< absl::string_view > > const &ins,int num_threads){
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePiecesAsImmutableProto, std::string,
+++ sentencepiece::ImmutableSentencePieceText);
+++ }
++ SWIGINTERN std::vector< std::vector< int > > sentencepiece_SentencePieceProcessor__NBestEncodeAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto idss = self->NBestEncodeAsIds(text, nbest_size);
++ for (auto &ids : idss) {
++@@ -3518,26 +3620,43 @@ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__NBes
++ add_bos, add_eos, reverse, emit_unk_piece);
++ return self->NBestEncodeAsSerializedProto(text, nbest_size);
++ }
++-SWIGINTERN std::vector< std::pair< std::vector< int >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float theta,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++SWIGINTERN sentencepiece::ImmutableNBestSentencePieceText sentencepiece_SentencePieceProcessor__NBestEncodeAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ RewriteIds(*self, static_cast<sentencepiece::ImmutableSentencePieceText *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return self->NBestEncodeAsImmutableProto(text, nbest_size);
+++ }
+++SWIGINTERN std::vector< std::pair< std::vector< int >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float alpha,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto idss = self->SampleEncodeAndScoreAsIds(text, num_samples,
++- theta, wor, include_best);
+++ alpha, wor, include_best);
++ for (auto &ids : idss) {
++ RewriteIds(*self, &ids.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return idss;
++ }
++-SWIGINTERN std::vector< std::pair< std::vector< std::string >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float theta,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++SWIGINTERN std::vector< std::pair< std::vector< std::string >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsPieces(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float alpha,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto piecess = self->SampleEncodeAndScoreAsPieces(text, num_samples,
++- theta, wor, include_best);
+++ alpha, wor, include_best);
++ for (auto &pieces : piecess) {
++ RewriteIds(*self, &pieces.first, add_bos, add_eos, reverse, emit_unk_piece);
++ }
++ return piecess;
++ }
++-SWIGINTERN float sentencepiece_SentencePieceProcessor__CalculateEntropy(sentencepiece::SentencePieceProcessor *self,absl::string_view text,float theta){
++- return self->CalculateEntropy(text, theta);
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float alpha,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ RewriteIds(*self, static_cast<sentencepiece::util::bytes *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return self->SampleEncodeAndScoreAsSerializedProto(text, num_samples,
+++ alpha, wor, include_best);
+++ }
+++SWIGINTERN sentencepiece::ImmutableNBestSentencePieceText sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float alpha,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
+++ RewriteIds(*self, static_cast<sentencepiece::util::bytes *>(nullptr),
+++ add_bos, add_eos, reverse, emit_unk_piece);
+++ return self->SampleEncodeAndScoreAsImmutableProto(text, num_samples,
+++ alpha, wor, include_best);
+++ }
+++SWIGINTERN float sentencepiece_SentencePieceProcessor__CalculateEntropy(sentencepiece::SentencePieceProcessor *self,absl::string_view text,float alpha){
+++ return self->CalculateEntropy(text, alpha);
++ }
++-SWIGINTERN std::vector< float > sentencepiece_SentencePieceProcessor__CalculateEntropyBatch(sentencepiece::SentencePieceProcessor *self,std::vector< absl::string_view > const &ins,float theta,int num_threads){
+++SWIGINTERN std::vector< float > sentencepiece_SentencePieceProcessor__CalculateEntropyBatch(sentencepiece::SentencePieceProcessor *self,std::vector< absl::string_view > const &ins,float alpha,int num_threads){
++ std::vector<float> outs(ins.size());
++ InitNumThreads(ins, &num_threads);
++ {
++@@ -3545,7 +3664,7 @@ SWIGINTERN std::vector< float > sentencepiece_SentencePieceProcessor__CalculateE
++ for (int n = 0; n < num_threads; ++n) {
++ pool.Schedule([&, n]() {
++ for (size_t i = n; i < ins.size(); i += num_threads) {
++- outs[i] = self->CalculateEntropy(ins[i], theta);
+++ outs[i] = self->CalculateEntropy(ins[i], alpha);
++ }
++ });
++ }
++@@ -3596,56 +3715,672 @@ SWIG_AsVal_unsigned_SS_long (PyObject *obj, unsigned long *val)
++ }
++ }
++ }
++-#endif
++- return SWIG_TypeError;
+++#endif
+++ return SWIG_TypeError;
+++}
+++
+++
+++SWIGINTERN int
+++SWIG_AsVal_unsigned_SS_int (PyObject * obj, unsigned int *val)
+++{
+++ unsigned long v;
+++ int res = SWIG_AsVal_unsigned_SS_long (obj, &v);
+++ if (SWIG_IsOK(res)) {
+++ if ((v > UINT_MAX)) {
+++ return SWIG_OverflowError;
+++ } else {
+++ if (val) *val = static_cast< unsigned int >(v);
+++ }
+++ }
+++ return res;
+++}
+++
+++SWIGINTERN void sentencepiece_SentencePieceTrainer__TrainFromString(absl::string_view arg){
+++ const auto _status = sentencepiece::SentencePieceTrainer::Train(arg);
+++ if (!_status.ok()) throw _status;
+++ return;
+++ }
+++SWIGINTERN void sentencepiece_SentencePieceTrainer__TrainFromMap(std::unordered_map< std::string,std::string > const &args){
+++ const auto _status = sentencepiece::SentencePieceTrainer::Train(args);
+++ if (!_status.ok()) throw _status;
+++ return;
+++ }
+++SWIGINTERN void sentencepiece_SentencePieceTrainer__TrainFromMap2(std::unordered_map< std::string,std::string > const &args,sentencepiece::SentenceIterator *iter){
+++ const auto _status = sentencepiece::SentencePieceTrainer::Train(args, iter);
+++ if (!_status.ok()) throw _status;
+++ return;
+++ }
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceTrainer__TrainFromMap3(std::unordered_map< std::string,std::string > const &args){
+++ sentencepiece::util::bytes model_proto;
+++ const auto _status = sentencepiece::SentencePieceTrainer::Train(args, nullptr, &model_proto);
+++ if (!_status.ok()) throw _status;
+++ return model_proto;
+++ }
+++SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceTrainer__TrainFromMap4(std::unordered_map< std::string,std::string > const &args,sentencepiece::SentenceIterator *iter){
+++ sentencepiece::util::bytes model_proto;
+++ const auto _status = sentencepiece::SentencePieceTrainer::Train(args, iter, &model_proto);
+++ if (!_status.ok()) throw _status;
+++ return model_proto;
+++ }
+++#ifdef __cplusplus
+++extern "C" {
+++#endif
+++SWIGINTERN PyObject *_wrap_new_ImmutableSentencePieceText_ImmutableSentencePiece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *result = 0 ;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "new_ImmutableSentencePieceText_ImmutableSentencePiece", 0, 0, 0)) SWIG_fail;
+++ {
+++ try {
+++ result = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *)new sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, SWIG_POINTER_NEW | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_delete_ImmutableSentencePieceText_ImmutableSentencePiece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, SWIG_POINTER_DISOWN | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "delete_ImmutableSentencePieceText_ImmutableSentencePiece" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
+++ {
+++ try {
+++ delete arg1;
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_Py_Void();
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_piece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ std::string *result = 0 ;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_piece" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
+++ {
+++ try {
+++ result = (std::string *) &((sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *)arg1)->piece();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = MakePyOutputString(*result, input_type);
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_surface(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ std::string *result = 0 ;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_surface" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
+++ {
+++ try {
+++ result = (std::string *) &((sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *)arg1)->surface();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = MakePyOutputString(*result, input_type);
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ uint32_t result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_id" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *)arg1)->id();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_unsigned_SS_int(static_cast< unsigned int >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_begin(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ uint32_t result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_begin" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *)arg1)->begin();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_unsigned_SS_int(static_cast< unsigned int >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_end(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ uint32_t result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_end" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *)arg1)->end();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_unsigned_SS_int(static_cast< unsigned int >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *ImmutableSentencePieceText_ImmutableSentencePiece_swigregister(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *obj;
+++ if (!SWIG_Python_UnpackTuple(args, "swigregister", 1, 1, &obj)) return NULL;
+++ SWIG_TypeNewClientData(SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, SWIG_NewClientData(obj));
+++ return SWIG_Py_Void();
+++}
+++
+++SWIGINTERN PyObject *ImmutableSentencePieceText_ImmutableSentencePiece_swiginit(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ return SWIG_Python_InitShadowInstance(args);
+++}
+++
+++SWIGINTERN PyObject *_wrap_new_ImmutableSentencePieceText(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *result = 0 ;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "new_ImmutableSentencePieceText", 0, 0, 0)) SWIG_fail;
+++ {
+++ try {
+++ result = (sentencepiece::ImmutableSentencePieceText *)new sentencepiece::ImmutableSentencePieceText();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_NEW | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_delete_ImmutableSentencePieceText(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_DISOWN | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "delete_ImmutableSentencePieceText" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ {
+++ try {
+++ delete arg1;
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_Py_Void();
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces_size(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ size_t result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_pieces_size" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableSentencePieceText const *)arg1)->pieces_size();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_size_t(static_cast< size_t >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_text(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ std::string *result = 0 ;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_text" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ {
+++ try {
+++ result = (std::string *) &((sentencepiece::ImmutableSentencePieceText const *)arg1)->text();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ PyObject *input_type = resultobj;
+++ resultobj = MakePyOutputString(*result, input_type);
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_score(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ float result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_score" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ {
+++ try {
+++ result = (float)((sentencepiece::ImmutableSentencePieceText const *)arg1)->score();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_float(static_cast< float >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_SerializeAsString(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ sentencepiece::util::bytes result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_SerializeAsString" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableSentencePieceText const *)arg1)->SerializeAsString();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = MakePyOutputBytes(result);
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ int arg2 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText_pieces", 2, 2, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "2"" of type '" "int""'");
+++ }
+++ arg2 = static_cast< int >(val2);
+++ {
+++ try {
+++ result = sentencepiece_ImmutableSentencePieceText_pieces((sentencepiece::ImmutableSentencePieceText const *)arg1,arg2);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece(static_cast< const sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *ImmutableSentencePieceText_swigregister(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *obj;
+++ if (!SWIG_Python_UnpackTuple(args, "swigregister", 1, 1, &obj)) return NULL;
+++ SWIG_TypeNewClientData(SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_NewClientData(obj));
+++ return SWIG_Py_Void();
+++}
+++
+++SWIGINTERN PyObject *ImmutableSentencePieceText_swiginit(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ return SWIG_Python_InitShadowInstance(args);
+++}
+++
+++SWIGINTERN PyObject *_wrap_new_ImmutableNBestSentencePieceText(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableNBestSentencePieceText *result = 0 ;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "new_ImmutableNBestSentencePieceText", 0, 0, 0)) SWIG_fail;
+++ {
+++ try {
+++ result = (sentencepiece::ImmutableNBestSentencePieceText *)new sentencepiece::ImmutableNBestSentencePieceText();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, SWIG_POINTER_NEW | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_delete_ImmutableNBestSentencePieceText(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, SWIG_POINTER_DISOWN | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "delete_ImmutableNBestSentencePieceText" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
+++ {
+++ try {
+++ delete arg1;
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_Py_Void();
+++ return resultobj;
+++fail:
+++ return NULL;
++ }
++
++
++-SWIGINTERN int
++-SWIG_AsVal_unsigned_SS_int (PyObject * obj, unsigned int *val)
++-{
++- unsigned long v;
++- int res = SWIG_AsVal_unsigned_SS_long (obj, &v);
++- if (SWIG_IsOK(res)) {
++- if ((v > UINT_MAX)) {
++- return SWIG_OverflowError;
++- } else {
++- if (val) *val = static_cast< unsigned int >(v);
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests_size(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ size_t result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_nbests_size" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableNBestSentencePieceText const *)arg1)->nbests_size();
+++ ReleaseResultObject(resultobj);
++ }
++- }
++- return res;
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_From_size_t(static_cast< size_t >(result));
+++ return resultobj;
+++fail:
+++ return NULL;
++ }
++
++-SWIGINTERN void sentencepiece_SentencePieceTrainer__TrainFromString(absl::string_view arg){
++- const auto _status = sentencepiece::SentencePieceTrainer::Train(arg);
++- if (!_status.ok()) throw _status;
++- return;
+++
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_SerializeAsString(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[1] ;
+++ sentencepiece::util::bytes result;
+++
+++ if (!args) SWIG_fail;
+++ swig_obj[0] = args;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_SerializeAsString" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
++ }
++-SWIGINTERN void sentencepiece_SentencePieceTrainer__TrainFromMap(std::unordered_map< std::string,std::string > const &args){
++- const auto _status = sentencepiece::SentencePieceTrainer::Train(args);
++- if (!_status.ok()) throw _status;
++- return;
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableNBestSentencePieceText const *)arg1)->SerializeAsString();
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
++ }
++-SWIGINTERN void sentencepiece_SentencePieceTrainer__TrainFromMap2(std::unordered_map< std::string,std::string > const &args,sentencepiece::SentenceIterator *iter){
++- const auto _status = sentencepiece::SentencePieceTrainer::Train(args, iter);
++- if (!_status.ok()) throw _status;
++- return;
+++ {
+++ resultobj = MakePyOutputBytes(result);
++ }
++-SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceTrainer__TrainFromMap3(std::unordered_map< std::string,std::string > const &args){
++- sentencepiece::util::bytes model_proto;
++- const auto _status = sentencepiece::SentencePieceTrainer::Train(args, nullptr, &model_proto);
++- if (!_status.ok()) throw _status;
++- return model_proto;
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
+++ int arg2 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::ImmutableSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText_nbests", 2, 2, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
++ }
++-SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceTrainer__TrainFromMap4(std::unordered_map< std::string,std::string > const &args,sentencepiece::SentenceIterator *iter){
++- sentencepiece::util::bytes model_proto;
++- const auto _status = sentencepiece::SentencePieceTrainer::Train(args, iter, &model_proto);
++- if (!_status.ok()) throw _status;
++- return model_proto;
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "2"" of type '" "int""'");
+++ }
+++ arg2 = static_cast< int >(val2);
+++ {
+++ try {
+++ result = sentencepiece_ImmutableNBestSentencePieceText_nbests((sentencepiece::ImmutableNBestSentencePieceText const *)arg1,arg2);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
++ }
++-#ifdef __cplusplus
++-extern "C" {
++-#endif
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText(static_cast< const sentencepiece::ImmutableSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *ImmutableNBestSentencePieceText_swigregister(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *obj;
+++ if (!SWIG_Python_UnpackTuple(args, "swigregister", 1, 1, &obj)) return NULL;
+++ SWIG_TypeNewClientData(SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, SWIG_NewClientData(obj));
+++ return SWIG_Py_Void();
+++}
+++
+++SWIGINTERN PyObject *ImmutableNBestSentencePieceText_swiginit(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ return SWIG_Python_InitShadowInstance(args);
+++}
+++
++ SWIGINTERN PyObject *_wrap_new_SentencePieceProcessor(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *result = 0 ;
++@@ -3992,165 +4727,16 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_CalculateEntropy__SWIG_0(PyObj
++ float *arg4 = (float *) 0 ;
++ void *argp1 = 0 ;
++ int res1 = 0 ;
++- float val3 ;
++- int ecode3 = 0 ;
++- void *argp4 = 0 ;
++- int res4 = 0 ;
++- sentencepiece::util::Status result;
++-
++- if ((nobjs < 4) || (nobjs > 4)) SWIG_fail;
++- res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++- if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++- }
++- arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = ustring.str();
++- }
++- ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "3"" of type '" "float""'");
++- }
++- arg3 = static_cast< float >(val3);
++- res4 = SWIG_ConvertPtr(swig_obj[3], &argp4,SWIGTYPE_p_float, 0 | 0 );
++- if (!SWIG_IsOK(res4)) {
++- SWIG_exception_fail(SWIG_ArgError(res4), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "4"" of type '" "float *""'");
++- }
++- arg4 = reinterpret_cast< float * >(argp4);
++- {
++- try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->CalculateEntropy(arg2,arg3,arg4);
++- ReleaseResultObject(resultobj);
++- }
++- catch (const sentencepiece::util::Status &status) {
++- SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++- }
++- }
++- {
++- if (!(&result)->ok()) {
++- SWIG_exception(ToSwigError((&result)->code()), (&result)->ToString().c_str());
++- }
++- resultobj = SWIG_From_bool((&result)->ok());
++- }
++- return resultobj;
++-fail:
++- return NULL;
++-}
++-
++-
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++- PyObject *resultobj = 0;
++- sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
++- bool arg5 ;
++- bool arg6 ;
++- void *argp1 = 0 ;
++- int res1 = 0 ;
++- int val3 ;
++- int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- bool val5 ;
++- int ecode5 = 0 ;
++- bool val6 ;
++- int ecode6 = 0 ;
++- PyObject *swig_obj[6] ;
++- std::vector< std::pair< std::vector< std::string >,float > > result;
++-
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAndScoreAsPieces", 6, 6, swig_obj)) SWIG_fail;
++- res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++- if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++- }
++- arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++- {
++- const PyInputString ustring(swig_obj[1]);
++- if (!ustring.IsAvalable()) {
++- PyErr_SetString(PyExc_TypeError, "not a string");
++- SWIG_fail;
++- }
++- resultobj = ustring.input_type();
++- arg2 = ustring.str();
++- }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++- if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "4"" of type '" "float""'");
++- }
++- arg4 = static_cast< float >(val4);
++- ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++- if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "5"" of type '" "bool""'");
++- }
++- arg5 = static_cast< bool >(val5);
++- ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++- if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsPieces" "', argument " "6"" of type '" "bool""'");
++- }
++- arg6 = static_cast< bool >(val6);
++- {
++- try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAndScoreAsPieces(arg2,arg3,arg4,arg5,arg6);
++- ReleaseResultObject(resultobj);
++- }
++- catch (const sentencepiece::util::Status &status) {
++- SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++- }
++- }
++- {
++- PyObject *input_type = resultobj;
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].first.size());
++- for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SET_ITEM(obj, j, MakePyOutputString(result[i].first[j], input_type));
++- }
++- PyList_SET_ITEM(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
++- }
++- }
++- return resultobj;
++-fail:
++- return NULL;
++-}
++-
++-
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++- PyObject *resultobj = 0;
++- sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++- absl::string_view arg2 ;
++- int arg3 ;
++- float arg4 ;
++- bool arg5 ;
++- bool arg6 ;
++- void *argp1 = 0 ;
++- int res1 = 0 ;
++- int val3 ;
+++ float val3 ;
++ int ecode3 = 0 ;
++- float val4 ;
++- int ecode4 = 0 ;
++- bool val5 ;
++- int ecode5 = 0 ;
++- bool val6 ;
++- int ecode6 = 0 ;
++- PyObject *swig_obj[6] ;
++- std::vector< std::pair< std::vector< int >,float > > result;
+++ void *argp4 = 0 ;
+++ int res4 = 0 ;
+++ sentencepiece::util::Status result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor_SampleEncodeAndScoreAsIds", 6, 6, swig_obj)) SWIG_fail;
+++ if ((nobjs < 4) || (nobjs > 4)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -4162,29 +4748,19 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyOb
++ resultobj = ustring.input_type();
++ arg2 = ustring.str();
++ }
++- ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ ecode3 = SWIG_AsVal_float(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "3"" of type '" "int""'");
++- }
++- arg3 = static_cast< int >(val3);
++- ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
++- if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "4"" of type '" "float""'");
++- }
++- arg4 = static_cast< float >(val4);
++- ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
++- if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "5"" of type '" "bool""'");
++- }
++- arg5 = static_cast< bool >(val5);
++- ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
++- if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor_SampleEncodeAndScoreAsIds" "', argument " "6"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "3"" of type '" "float""'");
++ }
++- arg6 = static_cast< bool >(val6);
+++ arg3 = static_cast< float >(val3);
+++ res4 = SWIG_ConvertPtr(swig_obj[3], &argp4,SWIGTYPE_p_float, 0 | 0 );
+++ if (!SWIG_IsOK(res4)) {
+++ SWIG_exception_fail(SWIG_ArgError(res4), "in method '" "SentencePieceProcessor_CalculateEntropy" "', argument " "4"" of type '" "float *""'");
+++ }
+++ arg4 = reinterpret_cast< float * >(argp4);
++ {
++ try {
++- result = ((sentencepiece::SentencePieceProcessor const *)arg1)->SampleEncodeAndScoreAsIds(arg2,arg3,arg4,arg5,arg6);
+++ result = ((sentencepiece::SentencePieceProcessor const *)arg1)->CalculateEntropy(arg2,arg3,arg4);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -4192,14 +4768,10 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds(PyOb
++ }
++ }
++ {
++- resultobj = PyList_New((&result)->size());
++- for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].first.size());
++- for (size_t j = 0; j < result[i].first.size(); ++j) {
++- PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i].first[j])));
++- }
++- PyList_SET_ITEM(resultobj, i, PyTuple_Pack(2, obj, PyFloat_FromDouble(static_cast<double>(result[i].second))));
+++ if (!(&result)->ok()) {
+++ SWIG_exception(ToSwigError((&result)->code()), (&result)->ToString().c_str());
++ }
+++ resultobj = SWIG_From_bool((&result)->ok());
++ }
++ return resultobj;
++ fail:
++@@ -5112,15 +5684,242 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProto(PyObj
++ }
++ }
++ {
++- resultobj = MakePyOutputBytes(result);
+++ resultobj = MakePyOutputBytes(result);
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsImmutableProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ absl::string_view arg2 ;
+++ bool arg3 ;
+++ int arg4 ;
+++ float arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ bool val3 ;
+++ int ecode3 = 0 ;
+++ int val4 ;
+++ int ecode4 = 0 ;
+++ float val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ PyObject *swig_obj[9] ;
+++ sentencepiece::ImmutableSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsImmutableProto", 9, 9, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = ustring.str();
+++ }
+++ ecode3 = SWIG_AsVal_bool(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "3"" of type '" "bool""'");
+++ }
+++ arg3 = static_cast< bool >(val3);
+++ ecode4 = SWIG_AsVal_int(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "4"" of type '" "int""'");
+++ }
+++ arg4 = static_cast< int >(val4);
+++ ecode5 = SWIG_AsVal_float(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "5"" of type '" "float""'");
+++ }
+++ arg5 = static_cast< float >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsImmutableProto" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsImmutableProto((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText(static_cast< const sentencepiece::ImmutableSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
+++ int arg3 ;
+++ bool arg4 ;
+++ int arg5 ;
+++ float arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ bool arg10 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ bool val4 ;
+++ int ecode4 = 0 ;
+++ int val5 ;
+++ int ecode5 = 0 ;
+++ float val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
+++ std::vector< std::vector< int > > result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsIdsBatch", 10, 10, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = ustring.str();
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "4"" of type '" "bool""'");
+++ }
+++ arg4 = static_cast< bool >(val4);
+++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "5"" of type '" "int""'");
+++ }
+++ arg5 = static_cast< int >(val5);
+++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "6"" of type '" "float""'");
+++ }
+++ arg6 = static_cast< float >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsIdsBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyObject *obj = PyList_New(result[i].size());
+++ for (size_t j = 0; j < result[i].size(); ++j) {
+++ PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ }
+++ PyList_SET_ITEM(resultobj, i, obj);
+++ }
+++ }
+++ {
+++ delete arg2;
++ }
++ return resultobj;
++ fail:
+++ {
+++ delete arg2;
+++ }
++ return NULL;
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ std::vector< absl::string_view > *arg2 = 0 ;
++@@ -5151,12 +5950,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SW
++ bool val10 ;
++ int ecode10 = 0 ;
++ PyObject *swig_obj[10] ;
++- std::vector< std::vector< int > > result;
+++ std::vector< std::vector< std::string > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsIdsBatch", 10, 10, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsPiecesBatch", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5182,47 +5981,47 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SW
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "3"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "3"" of type '" "int""'");
++ }
++ arg3 = static_cast< int >(val3);
++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
++ if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "4"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "4"" of type '" "bool""'");
++ }
++ arg4 = static_cast< bool >(val4);
++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
++ if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "5"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "5"" of type '" "int""'");
++ }
++ arg5 = static_cast< int >(val5);
++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
++ if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "6"" of type '" "float""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "6"" of type '" "float""'");
++ }
++ arg6 = static_cast< float >(val6);
++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
++ if (!SWIG_IsOK(ecode7)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "7"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "7"" of type '" "bool""'");
++ }
++ arg7 = static_cast< bool >(val7);
++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
++ if (!SWIG_IsOK(ecode8)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "8"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "8"" of type '" "bool""'");
++ }
++ arg8 = static_cast< bool >(val8);
++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
++ if (!SWIG_IsOK(ecode9)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "9"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "9"" of type '" "bool""'");
++ }
++ arg9 = static_cast< bool >(val9);
++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
++ if (!SWIG_IsOK(ecode10)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsIdsBatch" "', argument " "10"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "10"" of type '" "bool""'");
++ }
++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__EncodeAsIdsBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsPiecesBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5230,11 +6029,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsIdsBatch(PyObject *SW
++ }
++ }
++ {
+++ PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++ PyObject *obj = PyList_New(result[i].size());
++ for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SET_ITEM(obj, j, PyInt_FromLong(static_cast<long>(result[i][j])));
+++ PyList_SET_ITEM(obj, j, MakePyOutputString(result[i][j], input_type));
++ }
++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++@@ -5251,7 +6051,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ std::vector< absl::string_view > *arg2 = 0 ;
++@@ -5282,12 +6082,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject
++ bool val10 ;
++ int ecode10 = 0 ;
++ PyObject *swig_obj[10] ;
++- std::vector< std::vector< std::string > > result;
+++ BytesArray result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsPiecesBatch", 10, 10, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsSerializedProtoBatch", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5313,47 +6113,47 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "3"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "3"" of type '" "int""'");
++ }
++ arg3 = static_cast< int >(val3);
++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
++ if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "4"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "4"" of type '" "bool""'");
++ }
++ arg4 = static_cast< bool >(val4);
++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
++ if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "5"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "5"" of type '" "int""'");
++ }
++ arg5 = static_cast< int >(val5);
++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
++ if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "6"" of type '" "float""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "6"" of type '" "float""'");
++ }
++ arg6 = static_cast< float >(val6);
++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
++ if (!SWIG_IsOK(ecode7)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "7"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "7"" of type '" "bool""'");
++ }
++ arg7 = static_cast< bool >(val7);
++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
++ if (!SWIG_IsOK(ecode8)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "8"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "8"" of type '" "bool""'");
++ }
++ arg8 = static_cast< bool >(val8);
++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
++ if (!SWIG_IsOK(ecode9)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "9"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "9"" of type '" "bool""'");
++ }
++ arg9 = static_cast< bool >(val9);
++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
++ if (!SWIG_IsOK(ecode10)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsPiecesBatch" "', argument " "10"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "10"" of type '" "bool""'");
++ }
++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__EncodeAsPiecesBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5361,14 +6161,9 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsPiecesBatch(PyObject
++ }
++ }
++ {
++- PyObject *input_type = resultobj;
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyObject *obj = PyList_New(result[i].size());
++- for (size_t j = 0; j < result[i].size(); ++j) {
++- PyList_SET_ITEM(obj, j, MakePyOutputString(result[i][j], input_type));
++- }
++- PyList_SET_ITEM(resultobj, i, obj);
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
++ }
++ }
++ {
++@@ -5383,7 +6178,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsImmutableProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++ std::vector< absl::string_view > *arg2 = 0 ;
++@@ -5414,12 +6209,12 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(
++ bool val10 ;
++ int ecode10 = 0 ;
++ PyObject *swig_obj[10] ;
++- BytesArray result;
+++ SwigValueWrapper< std::vector< sentencepiece::ImmutableSentencePieceText > > result;
++
++- if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsSerializedProtoBatch", 10, 10, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__EncodeAsImmutableProtoBatch", 10, 10, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
++ {
++@@ -5445,47 +6240,47 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(
++ }
++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
++ if (!SWIG_IsOK(ecode3)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "3"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "3"" of type '" "int""'");
++ }
++ arg3 = static_cast< int >(val3);
++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
++ if (!SWIG_IsOK(ecode4)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "4"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "4"" of type '" "bool""'");
++ }
++ arg4 = static_cast< bool >(val4);
++ ecode5 = SWIG_AsVal_int(swig_obj[4], &val5);
++ if (!SWIG_IsOK(ecode5)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "5"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "5"" of type '" "int""'");
++ }
++ arg5 = static_cast< int >(val5);
++ ecode6 = SWIG_AsVal_float(swig_obj[5], &val6);
++ if (!SWIG_IsOK(ecode6)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "6"" of type '" "float""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "6"" of type '" "float""'");
++ }
++ arg6 = static_cast< float >(val6);
++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
++ if (!SWIG_IsOK(ecode7)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "7"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "7"" of type '" "bool""'");
++ }
++ arg7 = static_cast< bool >(val7);
++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
++ if (!SWIG_IsOK(ecode8)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "8"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "8"" of type '" "bool""'");
++ }
++ arg8 = static_cast< bool >(val8);
++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
++ if (!SWIG_IsOK(ecode9)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "9"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "9"" of type '" "bool""'");
++ }
++ arg9 = static_cast< bool >(val9);
++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
++ if (!SWIG_IsOK(ecode10)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsSerializedProtoBatch" "', argument " "10"" of type '" "bool""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__EncodeAsImmutableProtoBatch" "', argument " "10"" of type '" "bool""'");
++ }
++ arg10 = static_cast< bool >(val10);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__EncodeAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ result = sentencepiece_SentencePieceProcessor__EncodeAsImmutableProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -5495,7 +6290,8 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch(
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
+++ PyObject *obj = SWIG_NewPointerObj(new sentencepiece::ImmutableSentencePieceText((&result)->at(i)), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0);
+++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++ }
++ {
++@@ -5750,6 +6546,121 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsAsImmutableProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< int > *arg2 = 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::ImmutableSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodeIdsAsImmutableProto", 2, 2, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodeIdsAsImmutableProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<int> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<int>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyInt_Check(o)) {
+++ (*out)[i] = static_cast<int>(PyInt_AsLong(o));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__DecodeIdsAsImmutableProto((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< int > const &)*arg2);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText(static_cast< const sentencepiece::ImmutableSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ {
+++ delete arg2;
+++ }
+++ return resultobj;
+++fail:
+++ {
+++ delete arg2;
+++ }
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsImmutableProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< absl::string_view > *arg2 = 0 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::ImmutableSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodePiecesAsImmutableProto", 2, 2, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodePiecesAsImmutableProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<absl::string_view> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<absl::string_view>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ const PyInputString ustring(PyList_GetItem(swig_obj[1], i));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i] = ustring.str();
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsImmutableProto((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< absl::string_view > const &)*arg2);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText(static_cast< const sentencepiece::ImmutableSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ {
+++ delete arg2;
+++ }
+++ return resultobj;
+++fail:
+++ {
+++ delete arg2;
+++ }
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++@@ -6043,7 +6954,82 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ arg3 = static_cast< int >(val3);
++ {
++ try {
++- result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< absl::string_view > > const &)*arg2,arg3);
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< absl::string_view > > const &)*arg2,arg3);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
+++ }
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< std::vector< absl::string_view > > *arg2 = 0 ;
+++ int arg3 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ PyObject *swig_obj[3] ;
+++ SwigValueWrapper< std::vector< sentencepiece::ImmutableSentencePieceText > > result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch", 3, 3, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<std::vector<absl::string_view>> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<std::vector<absl::string_view>>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ const PyInputString ustring(PyList_GetItem(o, j));
+++ if (ustring.IsAvalable()) {
+++ (*out)[i][j] = ustring.str();
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"list must contain integers");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< absl::string_view > > const &)*arg2,arg3);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -6053,7 +7039,8 @@ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto
++ {
++ resultobj = PyList_New((&result)->size());
++ for (size_t i = 0; i < (&result)->size(); ++i) {
++- PyList_SET_ITEM(resultobj, i, MakePyOutputBytes(result[i]));
+++ PyObject *obj = SWIG_NewPointerObj(new sentencepiece::ImmutableSentencePieceText((&result)->at(i)), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0);
+++ PyList_SET_ITEM(resultobj, i, obj);
++ }
++ }
++ return resultobj;
++@@ -6323,6 +7310,86 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__NBestEncodeAsImmutableProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ absl::string_view arg2 ;
+++ int arg3 ;
+++ bool arg4 ;
+++ bool arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ bool val4 ;
+++ int ecode4 = 0 ;
+++ bool val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ PyObject *swig_obj[7] ;
+++ sentencepiece::ImmutableNBestSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__NBestEncodeAsImmutableProto", 7, 7, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__NBestEncodeAsImmutableProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = ustring.str();
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__NBestEncodeAsImmutableProto" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_bool(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__NBestEncodeAsImmutableProto" "', argument " "4"" of type '" "bool""'");
+++ }
+++ arg4 = static_cast< bool >(val4);
+++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__NBestEncodeAsImmutableProto" "', argument " "5"" of type '" "bool""'");
+++ }
+++ arg5 = static_cast< bool >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__NBestEncodeAsImmutableProto" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__NBestEncodeAsImmutableProto" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__NBestEncodeAsImmutableProto((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableNBestSentencePieceText(static_cast< const sentencepiece::ImmutableNBestSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++@@ -6550,6 +7617,216 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ absl::string_view arg2 ;
+++ int arg3 ;
+++ float arg4 ;
+++ bool arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ bool arg10 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ float val4 ;
+++ int ecode4 = 0 ;
+++ bool val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
+++ sentencepiece::util::bytes result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto", 10, 10, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = ustring.str();
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "4"" of type '" "float""'");
+++ }
+++ arg4 = static_cast< float >(val4);
+++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "5"" of type '" "bool""'");
+++ }
+++ arg5 = static_cast< bool >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = MakePyOutputBytes(result);
+++ }
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ absl::string_view arg2 ;
+++ int arg3 ;
+++ float arg4 ;
+++ bool arg5 ;
+++ bool arg6 ;
+++ bool arg7 ;
+++ bool arg8 ;
+++ bool arg9 ;
+++ bool arg10 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ float val4 ;
+++ int ecode4 = 0 ;
+++ bool val5 ;
+++ int ecode5 = 0 ;
+++ bool val6 ;
+++ int ecode6 = 0 ;
+++ bool val7 ;
+++ int ecode7 = 0 ;
+++ bool val8 ;
+++ int ecode8 = 0 ;
+++ bool val9 ;
+++ int ecode9 = 0 ;
+++ bool val10 ;
+++ int ecode10 = 0 ;
+++ PyObject *swig_obj[10] ;
+++ sentencepiece::ImmutableNBestSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto", 10, 10, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ const PyInputString ustring(swig_obj[1]);
+++ if (!ustring.IsAvalable()) {
+++ PyErr_SetString(PyExc_TypeError, "not a string");
+++ SWIG_fail;
+++ }
+++ resultobj = ustring.input_type();
+++ arg2 = ustring.str();
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ ecode4 = SWIG_AsVal_float(swig_obj[3], &val4);
+++ if (!SWIG_IsOK(ecode4)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode4), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "4"" of type '" "float""'");
+++ }
+++ arg4 = static_cast< float >(val4);
+++ ecode5 = SWIG_AsVal_bool(swig_obj[4], &val5);
+++ if (!SWIG_IsOK(ecode5)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode5), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "5"" of type '" "bool""'");
+++ }
+++ arg5 = static_cast< bool >(val5);
+++ ecode6 = SWIG_AsVal_bool(swig_obj[5], &val6);
+++ if (!SWIG_IsOK(ecode6)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode6), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "6"" of type '" "bool""'");
+++ }
+++ arg6 = static_cast< bool >(val6);
+++ ecode7 = SWIG_AsVal_bool(swig_obj[6], &val7);
+++ if (!SWIG_IsOK(ecode7)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode7), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "7"" of type '" "bool""'");
+++ }
+++ arg7 = static_cast< bool >(val7);
+++ ecode8 = SWIG_AsVal_bool(swig_obj[7], &val8);
+++ if (!SWIG_IsOK(ecode8)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode8), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "8"" of type '" "bool""'");
+++ }
+++ arg8 = static_cast< bool >(val8);
+++ ecode9 = SWIG_AsVal_bool(swig_obj[8], &val9);
+++ if (!SWIG_IsOK(ecode9)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode9), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "9"" of type '" "bool""'");
+++ }
+++ arg9 = static_cast< bool >(val9);
+++ ecode10 = SWIG_AsVal_bool(swig_obj[9], &val10);
+++ if (!SWIG_IsOK(ecode10)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode10), "in method '" "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto" "', argument " "10"" of type '" "bool""'");
+++ }
+++ arg10 = static_cast< bool >(val10);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto((sentencepiece::SentencePieceProcessor const *)arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableNBestSentencePieceText(static_cast< const sentencepiece::ImmutableNBestSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__CalculateEntropy(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++@@ -7009,6 +8286,31 @@ SWIGINTERN PyObject *SentencePieceTrainer_swigregister(PyObject *SWIGUNUSEDPARM(
++
++ static PyMethodDef SwigMethods[] = {
++ { "SWIG_PyInstanceMethod_New", SWIG_PyInstanceMethod_New, METH_O, NULL},
+++ { "new_ImmutableSentencePieceText_ImmutableSentencePiece", _wrap_new_ImmutableSentencePieceText_ImmutableSentencePiece, METH_NOARGS, NULL},
+++ { "delete_ImmutableSentencePieceText_ImmutableSentencePiece", _wrap_delete_ImmutableSentencePieceText_ImmutableSentencePiece, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_piece", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_piece, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_surface", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_surface, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_id", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_id, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_begin", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_begin, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_end", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_end, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_swigregister", ImmutableSentencePieceText_ImmutableSentencePiece_swigregister, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece_swiginit", ImmutableSentencePieceText_ImmutableSentencePiece_swiginit, METH_VARARGS, NULL},
+++ { "new_ImmutableSentencePieceText", _wrap_new_ImmutableSentencePieceText, METH_NOARGS, NULL},
+++ { "delete_ImmutableSentencePieceText", _wrap_delete_ImmutableSentencePieceText, METH_O, NULL},
+++ { "ImmutableSentencePieceText_pieces_size", _wrap_ImmutableSentencePieceText_pieces_size, METH_O, NULL},
+++ { "ImmutableSentencePieceText_text", _wrap_ImmutableSentencePieceText_text, METH_O, NULL},
+++ { "ImmutableSentencePieceText_score", _wrap_ImmutableSentencePieceText_score, METH_O, NULL},
+++ { "ImmutableSentencePieceText_SerializeAsString", _wrap_ImmutableSentencePieceText_SerializeAsString, METH_O, NULL},
+++ { "ImmutableSentencePieceText_pieces", _wrap_ImmutableSentencePieceText_pieces, METH_VARARGS, NULL},
+++ { "ImmutableSentencePieceText_swigregister", ImmutableSentencePieceText_swigregister, METH_O, NULL},
+++ { "ImmutableSentencePieceText_swiginit", ImmutableSentencePieceText_swiginit, METH_VARARGS, NULL},
+++ { "new_ImmutableNBestSentencePieceText", _wrap_new_ImmutableNBestSentencePieceText, METH_NOARGS, NULL},
+++ { "delete_ImmutableNBestSentencePieceText", _wrap_delete_ImmutableNBestSentencePieceText, METH_O, NULL},
+++ { "ImmutableNBestSentencePieceText_nbests_size", _wrap_ImmutableNBestSentencePieceText_nbests_size, METH_O, NULL},
+++ { "ImmutableNBestSentencePieceText_SerializeAsString", _wrap_ImmutableNBestSentencePieceText_SerializeAsString, METH_O, NULL},
+++ { "ImmutableNBestSentencePieceText_nbests", _wrap_ImmutableNBestSentencePieceText_nbests, METH_VARARGS, NULL},
+++ { "ImmutableNBestSentencePieceText_swigregister", ImmutableNBestSentencePieceText_swigregister, METH_O, NULL},
+++ { "ImmutableNBestSentencePieceText_swiginit", ImmutableNBestSentencePieceText_swiginit, METH_VARARGS, NULL},
++ { "new_SentencePieceProcessor", _wrap_new_SentencePieceProcessor, METH_NOARGS, NULL},
++ { "delete_SentencePieceProcessor", _wrap_delete_SentencePieceProcessor, METH_O, NULL},
++ { "SentencePieceProcessor_LoadFromSerializedProto", _wrap_SentencePieceProcessor_LoadFromSerializedProto, METH_VARARGS, NULL},
++@@ -7017,8 +8319,6 @@ static PyMethodDef SwigMethods[] = {
++ { "SentencePieceProcessor_SetVocabulary", _wrap_SentencePieceProcessor_SetVocabulary, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_ResetVocabulary", _wrap_SentencePieceProcessor_ResetVocabulary, METH_O, NULL},
++ { "SentencePieceProcessor_LoadVocabulary", _wrap_SentencePieceProcessor_LoadVocabulary, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_SampleEncodeAndScoreAsPieces", _wrap_SentencePieceProcessor_SampleEncodeAndScoreAsPieces, METH_VARARGS, NULL},
++- { "SentencePieceProcessor_SampleEncodeAndScoreAsIds", _wrap_SentencePieceProcessor_SampleEncodeAndScoreAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_CalculateEntropy", _wrap_SentencePieceProcessor_CalculateEntropy, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_GetPieceSize", _wrap_SentencePieceProcessor_GetPieceSize, METH_O, NULL},
++ { "SentencePieceProcessor_PieceToId", _wrap_SentencePieceProcessor_PieceToId, METH_VARARGS, NULL},
++@@ -7037,22 +8337,30 @@ static PyMethodDef SwigMethods[] = {
++ { "SentencePieceProcessor__EncodeAsIds", _wrap_SentencePieceProcessor__EncodeAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsPieces", _wrap_SentencePieceProcessor__EncodeAsPieces, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsSerializedProto", _wrap_SentencePieceProcessor__EncodeAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__EncodeAsImmutableProto", _wrap_SentencePieceProcessor__EncodeAsImmutableProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsIdsBatch", _wrap_SentencePieceProcessor__EncodeAsIdsBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsPiecesBatch", _wrap_SentencePieceProcessor__EncodeAsPiecesBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__EncodeAsSerializedProtoBatch", _wrap_SentencePieceProcessor__EncodeAsSerializedProtoBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__EncodeAsImmutableProtoBatch", _wrap_SentencePieceProcessor__EncodeAsImmutableProtoBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodeIds", _wrap_SentencePieceProcessor__DecodeIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePieces", _wrap_SentencePieceProcessor__DecodePieces, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodeIdsAsSerializedProto", _wrap_SentencePieceProcessor__DecodeIdsAsSerializedProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePiecesAsSerializedProto", _wrap_SentencePieceProcessor__DecodePiecesAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodeIdsAsImmutableProto", _wrap_SentencePieceProcessor__DecodeIdsAsImmutableProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodePiecesAsImmutableProto", _wrap_SentencePieceProcessor__DecodePiecesAsImmutableProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodeIdsBatch", _wrap_SentencePieceProcessor__DecodeIdsBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch", _wrap_SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePiecesBatch", _wrap_SentencePieceProcessor__DecodePiecesBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch", _wrap_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch", _wrap_SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__NBestEncodeAsIds", _wrap_SentencePieceProcessor__NBestEncodeAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__NBestEncodeAsPieces", _wrap_SentencePieceProcessor__NBestEncodeAsPieces, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__NBestEncodeAsSerializedProto", _wrap_SentencePieceProcessor__NBestEncodeAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__NBestEncodeAsImmutableProto", _wrap_SentencePieceProcessor__NBestEncodeAsImmutableProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__SampleEncodeAndScoreAsIds", _wrap_SentencePieceProcessor__SampleEncodeAndScoreAsIds, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__SampleEncodeAndScoreAsPieces", _wrap_SentencePieceProcessor__SampleEncodeAndScoreAsPieces, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto", _wrap_SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProto, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto", _wrap_SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__CalculateEntropy", _wrap_SentencePieceProcessor__CalculateEntropy, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__CalculateEntropyBatch", _wrap_SentencePieceProcessor__CalculateEntropyBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor_swigregister", SentencePieceProcessor_swigregister, METH_O, NULL},
++@@ -7076,6 +8384,9 @@ static PyMethodDef SwigMethods_proxydocs[] = {
++
++ static swig_type_info _swigt__p_char = {"_p_char", "char *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_float = {"_p_float", "float *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_sentencepiece__ImmutableNBestSentencePieceText = {"_p_sentencepiece__ImmutableNBestSentencePieceText", "sentencepiece::ImmutableNBestSentencePieceText *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_sentencepiece__ImmutableSentencePieceText = {"_p_sentencepiece__ImmutableSentencePieceText", "sentencepiece::ImmutableSentencePieceText *", 0, 0, (void*)0, 0};
+++static swig_type_info _swigt__p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece = {"_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece", "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_sentencepiece__SentenceIterator = {"_p_sentencepiece__SentenceIterator", "sentencepiece::SentenceIterator *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_sentencepiece__SentencePieceProcessor = {"_p_sentencepiece__SentencePieceProcessor", "sentencepiece::SentencePieceProcessor *", 0, 0, (void*)0, 0};
++ static swig_type_info _swigt__p_sentencepiece__SentencePieceTrainer = {"_p_sentencepiece__SentencePieceTrainer", "sentencepiece::SentencePieceTrainer *", 0, 0, (void*)0, 0};
++@@ -7089,6 +8400,9 @@ static swig_type_info _swigt__p_std__vectorT_std__vectorT_int_t_t = {"_p_std__ve
++ static swig_type_info *swig_type_initial[] = {
++ &_swigt__p_char,
++ &_swigt__p_float,
+++ &_swigt__p_sentencepiece__ImmutableNBestSentencePieceText,
+++ &_swigt__p_sentencepiece__ImmutableSentencePieceText,
+++ &_swigt__p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece,
++ &_swigt__p_sentencepiece__SentenceIterator,
++ &_swigt__p_sentencepiece__SentencePieceProcessor,
++ &_swigt__p_sentencepiece__SentencePieceTrainer,
++@@ -7102,6 +8416,9 @@ static swig_type_info *swig_type_initial[] = {
++
++ static swig_cast_info _swigc__p_char[] = { {&_swigt__p_char, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_float[] = { {&_swigt__p_float, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_sentencepiece__ImmutableNBestSentencePieceText[] = { {&_swigt__p_sentencepiece__ImmutableNBestSentencePieceText, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_sentencepiece__ImmutableSentencePieceText[] = { {&_swigt__p_sentencepiece__ImmutableSentencePieceText, 0, 0, 0},{0, 0, 0, 0}};
+++static swig_cast_info _swigc__p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece[] = { {&_swigt__p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_sentencepiece__SentenceIterator[] = { {&_swigt__p_sentencepiece__SentenceIterator, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_sentencepiece__SentencePieceProcessor[] = { {&_swigt__p_sentencepiece__SentencePieceProcessor, 0, 0, 0},{0, 0, 0, 0}};
++ static swig_cast_info _swigc__p_sentencepiece__SentencePieceTrainer[] = { {&_swigt__p_sentencepiece__SentencePieceTrainer, 0, 0, 0},{0, 0, 0, 0}};
++@@ -7115,6 +8432,9 @@ static swig_cast_info _swigc__p_std__vectorT_std__vectorT_int_t_t[] = { {&_swig
++ static swig_cast_info *swig_cast_initial[] = {
++ _swigc__p_char,
++ _swigc__p_float,
+++ _swigc__p_sentencepiece__ImmutableNBestSentencePieceText,
+++ _swigc__p_sentencepiece__ImmutableSentencePieceText,
+++ _swigc__p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece,
++ _swigc__p_sentencepiece__SentenceIterator,
++ _swigc__p_sentencepiece__SentencePieceProcessor,
++ _swigc__p_sentencepiece__SentencePieceTrainer,
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index 6c48bcd..2f2c84a 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -287,16 +287,44 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ ids2 = self.sp_.EncodeAsIds(text2)
++ pieces = self.sp_.EncodeAsPieces(text)
++ pieces2 = self.sp_.EncodeAsPieces(text2)
++- protos = self.sp_.EncodeAsSerializedProto(text)
++- proto2 = self.sp_.EncodeAsSerializedProto(text2)
+++ sprotos = self.sp_.EncodeAsSerializedProto(text)
+++ sproto2 = self.sp_.EncodeAsSerializedProto(text2)
+++ iprotos = self.sp_.EncodeAsImmutableProto(text)
+++ iprotos2 = self.sp_.EncodeAsImmutableProto(text2)
++
++ self.assertEqual(sp.encode(text, out_type=int), ids)
++ self.assertEqual(sp.encode(text, out_type=str), pieces)
++- self.assertEqual(sp.encode(text, out_type='proto'), protos)
+++ self.assertEqual(sp.encode(text, out_type='serialized_proto'), sprotos)
+++ self.assertEqual(sp.encode(text, out_type='immutable_proto'), iprotos)
++
++ self.assertEqual(sp.encode([text], out_type=int), [ids])
++ self.assertEqual(sp.encode([text], out_type=str), [pieces])
++- self.assertEqual(sp.encode([text], out_type='proto'), [protos])
+++ self.assertEqual(sp.encode([text], out_type='serialized_proto'), [sprotos])
+++ self.assertEqual(sp.encode([text], out_type='immutable_proto'), [iprotos])
+++
+++ self.assertEqual(len(iprotos), len(pieces))
+++ self.assertEqual(len(iprotos), len(ids))
+++ self.assertEqual(iprotos.text(), text)
+++
+++ self.assertEqual(len(iprotos2), len(pieces2))
+++ self.assertEqual(len(iprotos2), len(ids2))
+++ self.assertEqual(iprotos2.text(), text2)
+++
+++ for i in range(len(iprotos)):
+++ self.assertEqual(ids[i], iprotos.pieces(i).id())
+++ self.assertEqual(pieces[i], iprotos.pieces(i).piece())
+++
+++ for i, piece in enumerate(iprotos):
+++ self.assertEqual(ids[i], piece.id())
+++ self.assertEqual(pieces[i], piece.piece())
+++
+++ for i in range(len(iprotos2)):
+++ self.assertEqual(ids2[i], iprotos2.pieces(i).id())
+++ self.assertEqual(pieces2[i], iprotos2.pieces(i).piece())
+++
+++ for i, piece in enumerate(iprotos2):
+++ self.assertEqual(ids2[i], piece.id())
+++ self.assertEqual(pieces2[i], piece.piece())
++
++ detok_ids = self.sp_.DecodeIds(ids)
++ detok_pieces = self.sp_.DecodePieces(pieces)
++@@ -464,19 +492,29 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(d1, d4)
++ self.assertEqual(d1, d5)
++
++- r1 = sp.encode(texts, out_type='proto', num_threads=None)
++- r2 = sp.encode(texts, out_type='proto', num_threads=1)
++- r3 = sp.encode(texts, out_type='proto', num_threads=-1)
++- r4 = sp.encode(texts, out_type='proto', num_threads=8)
++- r5 = [sp.encode(s, out_type='proto') for s in texts]
+++ r1 = sp.encode(texts, out_type='serialized_proto', num_threads=None)
+++ r2 = sp.encode(texts, out_type='serialized_proto', num_threads=1)
+++ r3 = sp.encode(texts, out_type='serialized_proto', num_threads=-1)
+++ r4 = sp.encode(texts, out_type='serialized_proto', num_threads=8)
+++ r5 = [sp.encode(s, out_type='serialized_proto') for s in texts]
+++ self.assertEqual(r1, r2)
+++ self.assertEqual(r1, r3)
+++ self.assertEqual(r1, r4)
+++ self.assertEqual(r1, r5)
+++
+++ r1 = sp.encode(texts, out_type='immutable_proto', num_threads=None)
+++ r2 = sp.encode(texts, out_type='immutable_proto', num_threads=1)
+++ r3 = sp.encode(texts, out_type='immutable_proto', num_threads=-1)
+++ r4 = sp.encode(texts, out_type='immutable_proto', num_threads=8)
+++ r5 = [sp.encode(s, out_type='immutable_proto') for s in texts]
++ self.assertEqual(r1, r2)
++ self.assertEqual(r1, r3)
++ self.assertEqual(r1, r4)
++ self.assertEqual(r1, r5)
++
++- e1 = sp.calculate_entropy(texts, theta=1.0, num_threads=10)
++- e2 = sp.CalculateEntropy(texts, theta=1.0, num_threads=10)
++- e3 = [sp.calculate_entropy(s, theta=1.0) for s in texts]
+++ e1 = sp.calculate_entropy(texts, alpha=1.0, num_threads=10)
+++ e2 = sp.CalculateEntropy(texts, alpha=1.0, num_threads=10)
+++ e3 = [sp.calculate_entropy(s, alpha=1.0) for s in texts]
++ self.assertEqual(e1, e2)
++ self.assertEqual(e1, e3)
++
++diff --git a/src/sentencepiece_processor.cc b/src/sentencepiece_processor.cc
++index 805e0f9..482a45b 100644
++--- a/src/sentencepiece_processor.cc
+++++ b/src/sentencepiece_processor.cc
++@@ -54,65 +54,70 @@ std::vector<absl::string_view> ToPieceArray(const std::vector<std::string> &v) {
++ for (int i = 0; i < v.size(); ++i) out[i] = v[i];
++ return out;
++ }
+++
++ } // namespace
++
++-ImmutableSentencePieceText::ImmutableSentencePieceText() {}
++-ImmutableSentencePieceText::~ImmutableSentencePieceText() {}
+++ImmutableSentencePieceText::ImmutableSentencePieceText()
+++ : spt_(&SentencePieceText::default_instance()) {}
++
++ ImmutableSentencePieceText::ImmutableSentencePieceText(
++ const SentencePieceText &spt)
++ : spt_(&spt) {}
++
++-ImmutableSentencePieceText::ImmutableSentencePiece::ImmutableSentencePiece(
++- const SentencePieceText_SentencePiece &sp)
+++ImmutableSentencePieceText::~ImmutableSentencePieceText() {}
+++
+++ImmutableSentencePieceText_ImmutableSentencePiece::
+++ ImmutableSentencePieceText_ImmutableSentencePiece()
+++ : sp_(&SentencePieceText_SentencePiece::default_instance()) {}
+++
+++ImmutableSentencePieceText_ImmutableSentencePiece::
+++ ImmutableSentencePieceText_ImmutableSentencePiece(
+++ const SentencePieceText_SentencePiece &sp)
++ : sp_(&sp) {}
++
++-const std::string &ImmutableSentencePieceText::ImmutableSentencePiece::piece()
+++const std::string &ImmutableSentencePieceText_ImmutableSentencePiece::piece()
++ const {
++ return sp_->piece();
++ }
++
++-const std::string &ImmutableSentencePieceText::ImmutableSentencePiece::surface()
+++const std::string &ImmutableSentencePieceText_ImmutableSentencePiece::surface()
++ const {
++ return sp_->surface();
++ }
++
++-uint32_t ImmutableSentencePieceText::ImmutableSentencePiece::id() const {
+++uint32_t ImmutableSentencePieceText_ImmutableSentencePiece::id() const {
++ return sp_->id();
++ }
++
++-uint32_t ImmutableSentencePieceText::ImmutableSentencePiece::begin() const {
+++uint32_t ImmutableSentencePieceText_ImmutableSentencePiece::begin() const {
++ return sp_->begin();
++ }
++
++-uint32_t ImmutableSentencePieceText::ImmutableSentencePiece::end() const {
+++uint32_t ImmutableSentencePieceText_ImmutableSentencePiece::end() const {
++ return sp_->end();
++ }
++
++-std::vector<ImmutableSentencePieceText::ImmutableSentencePiece>
+++std::vector<ImmutableSentencePieceText_ImmutableSentencePiece>
++ ImmutableSentencePieceText::pieces() const {
++- std::vector<ImmutableSentencePieceText::ImmutableSentencePiece> pieces;
++- if (spt_ == nullptr) return pieces;
++- pieces.reserve(spt_->pieces_size());
+++ std::vector<ImmutableSentencePieceText_ImmutableSentencePiece> pieces(
+++ spt_->pieces_size());
++ for (int i = 0; i < spt_->pieces_size(); ++i)
++- pieces[i] = ImmutableSentencePiece(spt_->pieces(i));
+++ pieces[i] =
+++ ImmutableSentencePieceText_ImmutableSentencePiece(spt_->pieces(i));
++ return pieces;
++ }
++
++ size_t ImmutableSentencePieceText::pieces_size() const {
++- return spt_ ? spt_->pieces_size() : 0;
+++ return spt_->pieces_size();
++ }
++
++-ImmutableSentencePieceText::ImmutableSentencePiece
+++ImmutableSentencePieceText_ImmutableSentencePiece
++ ImmutableSentencePieceText::pieces(int index) const {
++- return ImmutableSentencePieceText::ImmutableSentencePiece(
++- spt_->pieces(index));
+++ return ImmutableSentencePieceText_ImmutableSentencePiece(spt_->pieces(index));
++ }
++
++ const std::string &ImmutableSentencePieceText::text() const {
++- if (spt_) return spt_->text();
++- static std::string *kEmptyString = new std::string();
++- return *kEmptyString;
+++ return spt_->text();
++ }
++
++ float ImmutableSentencePieceText::score() const {
++@@ -127,8 +132,8 @@ SentencePieceText *ImmutableSentencePieceText::mutable_proto() {
++ return rep_.get();
++ }
++
++-std::string ImmutableSentencePieceText::SerializeAsString() const {
++- return spt_ ? spt_->SerializeAsString() : "";
+++util::bytes ImmutableSentencePieceText::SerializeAsString() const {
+++ return spt_->SerializeAsString();
++ }
++
++ ImmutableNBestSentencePieceText::ImmutableNBestSentencePieceText() {}
++@@ -145,9 +150,8 @@ ImmutableSentencePieceText ImmutableNBestSentencePieceText::nbests(
++
++ std::vector<ImmutableSentencePieceText>
++ ImmutableNBestSentencePieceText::nbests() const {
++- std::vector<ImmutableSentencePieceText> nbests;
++- if (rep_ == nullptr) return nbests;
++- nbests.reserve(rep_->nbests_size());
+++ if (rep_ == nullptr) return {};
+++ std::vector<ImmutableSentencePieceText> nbests(rep_->nbests_size());
++ for (int i = 0; i < rep_->nbests_size(); ++i)
++ nbests[i] = ImmutableSentencePieceText(rep_->nbests(i));
++ return nbests;
++@@ -160,7 +164,7 @@ NBestSentencePieceText *ImmutableNBestSentencePieceText::mutable_proto() {
++ return rep_.get();
++ }
++
++-std::string ImmutableNBestSentencePieceText::SerializeAsString() const {
+++util::bytes ImmutableNBestSentencePieceText::SerializeAsString() const {
++ return rep_ ? rep_->SerializeAsString() : "";
++ }
++
++@@ -1044,8 +1048,35 @@ std::string SentencePieceProcessor::serialized_model_proto() const {
++ // std::random_device.
++ void SetRandomGeneratorSeed(unsigned int seed);
++
++-namespace io {
+++void ConvertToUnicodeSpans(SentencePieceText *spt) {
+++ if (spt == nullptr) return;
+++
+++ std::vector<int> utf8_to_unicode(spt->text().size() + 1, 0);
+++ absl::string_view str = spt->text();
+++ size_t prev = 0;
+++ int ulen = 0;
+++ while (!str.empty()) {
+++ const size_t mblen = string_util::OneCharLen(str.data());
+++ for (int i = prev; i < prev + mblen; ++i) {
+++ utf8_to_unicode[i] = ulen;
+++ }
+++ ++ulen;
+++ prev += mblen;
+++ str.remove_prefix(mblen);
+++ }
+++ utf8_to_unicode[prev] = ulen;
+++
+++ auto clip = [&](int s) {
+++ return std::min<int>(std::max<int>(0, s), utf8_to_unicode.size() - 1);
+++ };
++
+++ for (auto &piece : *(spt->mutable_pieces())) {
+++ piece.set_begin(utf8_to_unicode[clip(piece.begin())]);
+++ piece.set_end(utf8_to_unicode[clip(piece.end())]);
+++ }
+++}
+++
+++namespace io {
++ util::Status LoadModelProto(absl::string_view filename,
++ ModelProto *model_proto) {
++ if (filename.empty()) {
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index 8124c59..b7fae6a 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -157,35 +157,39 @@ class SentencePieceText_SentencePiece;
++ // This wrapper only allows an immutable access to the proto and
++ // hides the actual implementation of protobuf.
++ // See sentencepiece.proto for the details of this class.
+++class ImmutableSentencePieceText_ImmutableSentencePiece {
+++ public:
+++ ImmutableSentencePieceText_ImmutableSentencePiece();
+++ ~ImmutableSentencePieceText_ImmutableSentencePiece() = default;
+++
+++ const std::string &piece() const;
+++ const std::string &surface() const;
+++ uint32_t id() const;
+++ uint32_t begin() const;
+++ uint32_t end() const;
+++
+++ friend class ImmutableSentencePieceText;
+++
+++ private:
+++ explicit ImmutableSentencePieceText_ImmutableSentencePiece(
+++ const SentencePieceText_SentencePiece &sp);
+++ const SentencePieceText_SentencePiece *sp_ = nullptr;
+++};
+++
++ class ImmutableSentencePieceText {
++ public:
++ ImmutableSentencePieceText();
++ virtual ~ImmutableSentencePieceText();
++
++- class ImmutableSentencePiece {
++- public:
++- ~ImmutableSentencePiece() = default;
++- const std::string &piece() const;
++- const std::string &surface() const;
++- uint32_t id() const;
++- uint32_t begin() const;
++- uint32_t end() const;
+++ std::vector<ImmutableSentencePieceText_ImmutableSentencePiece> pieces() const;
++
++- friend class ImmutableSentencePieceText;
++-
++- private:
++- ImmutableSentencePiece() = default;
++- explicit ImmutableSentencePiece(const SentencePieceText_SentencePiece &sp);
++- const SentencePieceText_SentencePiece *sp_ = nullptr;
++- };
++-
++- std::vector<ImmutableSentencePiece> pieces() const;
++ size_t pieces_size() const;
++- ImmutableSentencePiece pieces(int index) const;
+++ ImmutableSentencePieceText_ImmutableSentencePiece pieces(int index) const;
+++
++ const std::string &text() const;
++ float score() const;
++
++- std::string SerializeAsString() const;
+++ util::bytes SerializeAsString() const;
++
++ // Returns the actual mutable proto.
++ // Do not use this outside of SentencePieceProcessor, as
++@@ -214,7 +218,7 @@ class ImmutableNBestSentencePieceText {
++ size_t nbests_size() const;
++ ImmutableSentencePieceText nbests(int index) const;
++
++- std::string SerializeAsString() const;
+++ util::bytes SerializeAsString() const;
++
++ // Returns the actual mutable proto.
++ // Do not use this outside of SentencePieceProcessor, as
++@@ -398,7 +402,7 @@ class SentencePieceProcessor {
++ float alpha, SentencePieceText *spt) const;
++
++ virtual util::Status SampleEncodeAndScore(
++- absl::string_view input, int samples, float alpha, bool wor,
+++ absl::string_view input, int num_samples, float alpha, bool wor,
++ bool include_best, NBestSentencePieceText *samples_spt) const;
++
++ // DEPRECATED: Remove this API and use std::vector<std::string_view>
++@@ -534,11 +538,11 @@ class SentencePieceProcessor {
++ }
++
++ virtual util::bytes SampleEncodeAndScoreAsSerializedProto(
++- absl::string_view input, int samples, float alpha, bool wor,
++- bool include_best, int nbest_size) const {
+++ absl::string_view input, int num_samples, float alpha, bool wor,
+++ bool include_best) const {
++ DEFINE_SPP_SERIALIZED_PROTO_IMPL(SampleEncodeAndScore,
++ ImmutableNBestSentencePieceText, input,
++- samples, alpha, wor, include_best);
+++ num_samples, alpha, wor, include_best);
++ }
++
++ // TODO(taku): Remove this API and use std::vector<std::string_view>
++@@ -579,11 +583,11 @@ class SentencePieceProcessor {
++ }
++
++ virtual ImmutableNBestSentencePieceText SampleEncodeAndScoreAsImmutableProto(
++- absl::string_view input, int samples, float alpha, bool wor,
++- bool include_best, int nbest_size) const {
+++ absl::string_view input, int num_samples, float alpha, bool wor,
+++ bool include_best) const {
++ DEFINE_SPP_IMMUTABLE_PROTO_IMPL(SampleEncodeAndScore,
++ ImmutableNBestSentencePieceText, input,
++- samples, alpha, wor, include_best);
+++ num_samples, alpha, wor, include_best);
++ }
++
++ // TODO(taku): Remove this API and use std::vector<std::string_view>
++@@ -703,6 +707,9 @@ class SentencePieceProcessor {
++ // std::random_device.
++ void SetRandomGeneratorSeed(unsigned int seed);
++
+++// Converts the utf8 byte spans into Unicode char span.
+++void ConvertToUnicodeSpans(SentencePieceText *spt);
+++
++ #ifndef SWIG
++ // IO related functions to absorb model formats.
++ namespace io {
++diff --git a/src/sentencepiece_processor_test.cc b/src/sentencepiece_processor_test.cc
++index ed651f7..ff55aeb 100644
++--- a/src/sentencepiece_processor_test.cc
+++++ b/src/sentencepiece_processor_test.cc
++@@ -1564,6 +1564,10 @@ TEST(SentencePieceProcessorTest, VocabularyTest) {
++
++ TEST(SentencePieceProcessorTest, ImmutableSentencePieceTextTest) {
++ ImmutableSentencePieceText spt;
+++ EXPECT_TRUE(spt.text().empty());
+++ EXPECT_EQ(spt.score(), 0.0);
+++ EXPECT_TRUE(spt.SerializeAsString().empty());
+++
++ auto *v = spt.mutable_proto();
++
++ v->set_text("hello world");
++@@ -1586,52 +1590,123 @@ TEST(SentencePieceProcessorTest, ImmutableSentencePieceTextTest) {
++ EXPECT_EQ(v->pieces(i).end(), spt.pieces(i).end());
++ }
++
++- int n = 0;
++- for (auto &p : spt.pieces()) {
++- EXPECT_EQ(v->pieces(n).surface(), p.surface());
++- EXPECT_EQ(v->pieces(n).piece(), p.piece());
++- EXPECT_EQ(v->pieces(n).id(), p.id());
++- EXPECT_EQ(v->pieces(n).begin(), p.begin());
++- EXPECT_EQ(v->pieces(n).end(), p.end());
++- ++n;
++- }
++-
++- EXPECT_EQ(v->text(), spt.text());
++- EXPECT_EQ(v->score(), spt.score());
++- EXPECT_EQ(v->SerializeAsString(), spt.SerializeAsString());
+++ auto check_proto = [&v](const ImmutableSentencePieceText &s) {
+++ int n = 0;
+++ for (auto &p : s.pieces()) {
+++ EXPECT_EQ(v->pieces(n).surface(), p.surface());
+++ EXPECT_EQ(v->pieces(n).piece(), p.piece());
+++ EXPECT_EQ(v->pieces(n).id(), p.id());
+++ EXPECT_EQ(v->pieces(n).begin(), p.begin());
+++ EXPECT_EQ(v->pieces(n).end(), p.end());
+++ ++n;
+++ }
+++ EXPECT_EQ(v->text(), s.text());
+++ EXPECT_EQ(v->score(), s.score());
+++ EXPECT_EQ(v->SerializeAsString(), s.SerializeAsString());
+++ };
++
++ // test copy.
++- auto spt2 = spt;
++- EXPECT_EQ(spt2.pieces_size(), spt.pieces_size());
++- for (int i = 0; i < spt.pieces_size(); ++i) {
++- EXPECT_EQ(spt2.pieces(i).surface(), spt.pieces(i).surface());
++- EXPECT_EQ(spt2.pieces(i).piece(), spt.pieces(i).piece());
++- EXPECT_EQ(spt2.pieces(i).id(), spt.pieces(i).id());
++- EXPECT_EQ(spt2.pieces(i).begin(), spt.pieces(i).begin());
++- EXPECT_EQ(spt2.pieces(i).end(), spt.pieces(i).end());
++- }
+++ const auto spt2 = spt;
+++ check_proto(spt2);
+++
+++ // test assign.
+++ const ImmutableSentencePieceText spt3(spt);
+++ check_proto(spt3);
+++
+++ // default piece.
+++ const ImmutableSentencePieceText_ImmutableSentencePiece piece;
+++ EXPECT_TRUE(piece.surface().empty());
+++ EXPECT_TRUE(piece.piece().empty());
+++ EXPECT_EQ(piece.begin(), 0);
+++ EXPECT_EQ(piece.end(), 0);
+++ EXPECT_EQ(piece.id(), 0);
++ }
++
++ TEST(SentencePieceProcessorTest, ImmutableNBestSentencePieceTextTest) {
++ ImmutableNBestSentencePieceText spt;
+++ EXPECT_EQ(spt.nbests_size(), 0);
+++ EXPECT_TRUE(spt.SerializeAsString().empty());
+++
++ auto *v = spt.mutable_proto();
+++
++ for (int i = 0; i < 10; ++i) {
++ auto *p = v->add_nbests();
++ p->set_text(absl::StrCat("text_", i));
++ p->set_score(2.0 * i);
++ }
++
++- EXPECT_EQ(v->nbests_size(), spt.nbests_size());
++- for (int i = 0; i < v->nbests_size(); ++i) {
++- EXPECT_EQ(v->nbests(i).text(), spt.nbests(i).text());
++- EXPECT_EQ(v->nbests(i).score(), spt.nbests(i).score());
++- }
++- EXPECT_EQ(v->SerializeAsString(), spt.SerializeAsString());
+++ auto check_proto = [&v](const ImmutableNBestSentencePieceText &s) {
+++ EXPECT_EQ(v->nbests_size(), s.nbests_size());
+++ for (int i = 0; i < v->nbests_size(); ++i) {
+++ EXPECT_EQ(v->nbests(i).text(), s.nbests(i).text());
+++ EXPECT_EQ(v->nbests(i).score(), s.nbests(i).score());
+++ }
+++ EXPECT_EQ(v->SerializeAsString(), s.SerializeAsString());
+++ };
+++
+++ check_proto(spt);
++
++ // test copy.
++- auto spt2 = spt;
++- EXPECT_EQ(spt2.nbests_size(), spt.nbests_size());
++- EXPECT_EQ(spt2.SerializeAsString(), spt.SerializeAsString());
+++ const auto spt2 = spt;
+++ check_proto(spt2);
+++
+++ // test assign.
+++ const ImmutableNBestSentencePieceText spt3(spt);
+++ check_proto(spt3);
+++}
+++
+++TEST(SentencePieceProcessorTest, ConvertToUnicodeSpansTest) {
+++ auto make_spt = [&](const std::vector<std::string> &tokens) {
+++ SentencePieceText spt;
+++ int prev = 0;
+++ std::string text;
+++ for (const auto &tok : tokens) {
+++ auto *piece = spt.add_pieces();
+++ piece->set_surface(tok);
+++ piece->set_piece(tok);
+++ piece->set_begin(prev);
+++ piece->set_end(prev + tok.size());
+++ prev += tok.size();
+++ text += tok;
+++ }
+++ spt.set_text(text);
+++ ConvertToUnicodeSpans(&spt);
+++ return spt;
+++ };
+++
+++ {
+++ const auto spt = make_spt({"hello", "_world", "."});
+++ EXPECT_EQ(spt.pieces_size(), 3);
+++ EXPECT_EQ(spt.pieces(0).begin(), 0);
+++ EXPECT_EQ(spt.pieces(0).end(), 5);
+++ EXPECT_EQ(spt.pieces(1).begin(), 5);
+++ EXPECT_EQ(spt.pieces(1).end(), 11);
+++ EXPECT_EQ(spt.pieces(2).begin(), 11);
+++ EXPECT_EQ(spt.pieces(2).end(), 12);
+++ }
+++
+++ {
+++ const auto spt = make_spt({"これは", "test", "です"});
+++ EXPECT_EQ(spt.pieces_size(), 3);
+++ EXPECT_EQ(spt.pieces(0).begin(), 0);
+++ EXPECT_EQ(spt.pieces(0).end(), 3);
+++ EXPECT_EQ(spt.pieces(1).begin(), 3);
+++ EXPECT_EQ(spt.pieces(1).end(), 7);
+++
+++ EXPECT_EQ(spt.pieces(2).begin(), 7);
+++ EXPECT_EQ(spt.pieces(2).end(), 9);
+++ }
+++
+++ {
+++ const auto spt = make_spt({"いABは", "にほCD", "へと"});
+++ EXPECT_EQ(spt.pieces_size(), 3);
+++ EXPECT_EQ(spt.pieces(0).begin(), 0);
+++ EXPECT_EQ(spt.pieces(0).end(), 4);
+++ EXPECT_EQ(spt.pieces(1).begin(), 4);
+++ EXPECT_EQ(spt.pieces(1).end(), 8);
+++ EXPECT_EQ(spt.pieces(2).begin(), 8);
+++ EXPECT_EQ(spt.pieces(2).end(), 10);
+++ }
++ }
++
++ } // namespace sentencepiece
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 3 Aug 2022 02:24:53 +0900
++Subject: Adds more unittests
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/__init__.py | 48 +++-
++ python/src/sentencepiece/sentencepiece.i | 45 +++-
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 213 +++++++++++++++--
++ python/test/sentencepiece_test.py | 301 ++++++++++++++++--------
++ src/sentencepiece_processor.cc | 67 +++---
++ src/sentencepiece_processor.h | 19 +-
++ src/sentencepiece_processor_test.cc | 11 +-
++ 7 files changed, 532 insertions(+), 172 deletions(-)
++
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index 69a9825..07acb94 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -98,6 +98,9 @@ class ImmutableSentencePieceText(object):
++ def pieces_size(self):
++ return _sentencepiece.ImmutableSentencePieceText_pieces_size(self)
++
+++ def pieces(self, index):
+++ return _sentencepiece.ImmutableSentencePieceText_pieces(self, index)
+++
++ def text(self):
++ return _sentencepiece.ImmutableSentencePieceText_text(self)
++
++@@ -107,18 +110,24 @@ class ImmutableSentencePieceText(object):
++ def SerializeAsString(self):
++ return _sentencepiece.ImmutableSentencePieceText_SerializeAsString(self)
++
++- def pieces(self, index):
++- return _sentencepiece.ImmutableSentencePieceText_pieces(self, index)
+++ def _pieces(self, index):
+++ return _sentencepiece.ImmutableSentencePieceText__pieces(self, index)
+++
+++ def pieces(self, i):
+++ return self._pieces(i)
++
++ def __len__(self):
++ return self.pieces_size()
++
++ def __getitem__(self, i):
++- return self.pieces(i)
+++ return self._pieces(i)
++
++ def __eq__(self, other):
++ return self.SerializeAsString() == other.SerializeAsString()
++
+++ def __hash__(self):
+++ return hash(self.SerializeAsString())
+++
++
++ # Register ImmutableSentencePieceText in _sentencepiece:
++ _sentencepiece.ImmutableSentencePieceText_swigregister(ImmutableSentencePieceText)
++@@ -134,21 +143,30 @@ class ImmutableNBestSentencePieceText(object):
++ def nbests_size(self):
++ return _sentencepiece.ImmutableNBestSentencePieceText_nbests_size(self)
++
+++ def nbests(self, index):
+++ return _sentencepiece.ImmutableNBestSentencePieceText_nbests(self, index)
+++
++ def SerializeAsString(self):
++ return _sentencepiece.ImmutableNBestSentencePieceText_SerializeAsString(self)
++
++- def nbests(self, index):
++- return _sentencepiece.ImmutableNBestSentencePieceText_nbests(self, index)
+++ def _nbests(self, index):
+++ return _sentencepiece.ImmutableNBestSentencePieceText__nbests(self, index)
+++
+++ def __nbests__(self, i):
+++ return self._nbests(i)
++
++ def __len__(self):
++ return self.nbests_size()
++
++ def __getitem__(self, i):
++- return self.nbests(i)
+++ return self._nbests(i)
++
++ def __eq__(self, other):
++ return self.SerializeAsString() == other.SerializeAsString()
++
+++ def __hash__(self):
+++ return hash(self.SerializeAsString())
+++
++
++ # Register ImmutableNBestSentencePieceText in _sentencepiece:
++ _sentencepiece.ImmutableNBestSentencePieceText_swigregister(ImmutableNBestSentencePieceText)
++@@ -272,6 +290,9 @@ class SentencePieceProcessor(object):
++ def _DecodeIdsAsSerializedProtoBatch(self, ins, num_threads):
++ return _sentencepiece.SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch(self, ins, num_threads)
++
+++ def _DecodeIdsAsImmutableProtoBatch(self, ins, num_threads):
+++ return _sentencepiece.SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch(self, ins, num_threads)
+++
++ def _DecodePiecesBatch(self, ins, num_threads):
++ return _sentencepiece.SentencePieceProcessor__DecodePiecesBatch(self, ins, num_threads)
++
++@@ -539,6 +560,8 @@ class SentencePieceProcessor(object):
++ return self._NBestEncodeAsImmutableProto(text, nbest_size,
++ add_bos, add_eos, reverse, emit_unk_piece)
++
+++ raise RuntimeError('unknown out_type')
+++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++
++@@ -621,10 +644,21 @@ class SentencePieceProcessor(object):
++ if out_type is int:
++ return self._SampleEncodeAndScoreAsIds(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++- else:
+++ if out_type is str:
++ return self._SampleEncodeAndScoreAsPieces(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++
+++ if out_type == 'serialized_proto' or out_type == 'proto':
+++ return self._SampleEncodeAndScoreAsSerializedProto(text, num_samples, alpha, wor, include_best,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ if out_type == 'immutable_proto':
+++ return self._SampleEncodeAndScoreAsImmutableProto(text, num_samples, alpha, wor, include_best,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ raise RuntimeError('unknown output type')
+++
+++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 1e2e1e0..f3a4f30 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -2,6 +2,7 @@
++ %include exception.i
++
++ %{
+++
++ #include <iostream>
++ #include <algorithm>
++ #include <functional>
++@@ -286,8 +287,10 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ %ignore sentencepiece::SentencePieceProcessor::status;
++ %ignore sentencepiece::ImmutableSentencePieceText::mutable_proto;
++ %ignore sentencepiece::ImmutableSentencePieceText::pieces() const;
+++%ignore sentencepiece::ImmutableSentencePieceText::ConvertToUnicodeSpans;
++ %ignore sentencepiece::ImmutableNBestSentencePieceText::mutable_proto;
++ %ignore sentencepiece::ImmutableNBestSentencePieceText::nbests() const;
+++%ignore sentencepiece::ImmutableNBestSentencePieceText::ConvertToUnicodeSpans;
++
++ %ignore sentencepiece::SentencePieceProcessor::Encode;
++ %ignore sentencepiece::SentencePieceProcessor::SampleEncode;
++@@ -481,6 +484,13 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ sentencepiece::util::bytes);
++ }
++
+++ std::vector<sentencepiece::ImmutableSentencePieceText>
+++ _DecodeIdsAsImmutableProtoBatch(
+++ const std::vector<std::vector<int>> &ins, int num_threads) const {
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIdsAsImmutableProto, int,
+++ sentencepiece::ImmutableSentencePieceText);
+++ }
+++
++ std::vector<std::string> _DecodePiecesBatch(
++ const std::vector<std::vector<absl::string_view>> &ins, int num_threads) const {
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePieces, std::string, std::string);
++@@ -852,6 +862,8 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return self._NBestEncodeAsImmutableProto(text, nbest_size,
++ add_bos, add_eos, reverse, emit_unk_piece)
++
+++ raise RuntimeError('unknown out_type')
+++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++
++@@ -934,10 +946,21 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ if out_type is int:
++ return self._SampleEncodeAndScoreAsIds(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++- else:
+++ if out_type is str:
++ return self._SampleEncodeAndScoreAsPieces(text, num_samples, alpha, wor, include_best,
++ add_bos, add_eos, reverse, emit_unk_piece)
++
+++ if out_type == 'serialized_proto' or out_type == 'proto':
+++ return self._SampleEncodeAndScoreAsSerializedProto(text, num_samples, alpha, wor, include_best,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ if out_type == 'immutable_proto':
+++ return self._SampleEncodeAndScoreAsImmutableProto(text, num_samples, alpha, wor, include_best,
+++ add_bos, add_eos, reverse, emit_unk_piece)
+++
+++ raise RuntimeError('unknown output type')
+++
+++
++ if type(input) is list:
++ return [_encode(n) for n in input]
++
++@@ -1187,7 +1210,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++
++ %extend sentencepiece::ImmutableSentencePieceText {
++- ImmutableSentencePieceText_ImmutableSentencePiece pieces(int index) const {
+++ ImmutableSentencePieceText_ImmutableSentencePiece _pieces(int index) const {
++ if (index < 0 || index >= static_cast<int>($self->pieces_size())) {
++ throw sentencepiece::util::Status(
++ sentencepiece::util::StatusCode::kOutOfRange,
++@@ -1197,19 +1220,25 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++
++ %pythoncode {
+++ def pieces(self, i):
+++ return self._pieces(i)
+++
++ def __len__(self):
++ return self.pieces_size()
++
++ def __getitem__(self, i):
++- return self.pieces(i)
+++ return self._pieces(i)
++
++ def __eq__(self, other):
++ return self.SerializeAsString() == other.SerializeAsString()
+++
+++ def __hash__(self):
+++ return hash(self.SerializeAsString())
++ }
++ }
++
++ %extend sentencepiece::ImmutableNBestSentencePieceText {
++- ImmutableSentencePieceText nbests(int index) const {
+++ ImmutableSentencePieceText _nbests(int index) const {
++ if (index < 0 || index >= static_cast<int>($self->nbests_size())) {
++ throw sentencepiece::util::Status(
++ sentencepiece::util::StatusCode::kOutOfRange,
++@@ -1219,14 +1248,20 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++
++ %pythoncode {
+++ def __nbests__(self, i):
+++ return self._nbests(i)
+++
++ def __len__(self):
++ return self.nbests_size()
++
++ def __getitem__(self, i):
++- return self.nbests(i)
+++ return self._nbests(i)
++
++ def __eq__(self, other):
++ return self.SerializeAsString() == other.SerializeAsString()
+++
+++ def __hash__(self):
+++ return hash(self.SerializeAsString())
++ }
++ }
++
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 9776b0f..22e0708 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -2811,6 +2811,7 @@ namespace swig {
++ }
++
++
+++
++ #include <iostream>
++ #include <algorithm>
++ #include <functional>
++@@ -3132,16 +3133,6 @@ SWIG_From_size_t (size_t value)
++ }
++
++
++- #define SWIG_From_double PyFloat_FromDouble
++-
++-
++-SWIGINTERNINLINE PyObject *
++-SWIG_From_float (float value)
++-{
++- return SWIG_From_double (value);
++-}
++-
++-
++ SWIGINTERN int
++ SWIG_AsVal_double (PyObject *obj, double *val)
++ {
++@@ -3282,7 +3273,17 @@ SWIG_AsVal_int (PyObject * obj, int *val)
++ return res;
++ }
++
++-SWIGINTERN sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece sentencepiece_ImmutableSentencePieceText_pieces(sentencepiece::ImmutableSentencePieceText const *self,int index){
+++
+++ #define SWIG_From_double PyFloat_FromDouble
+++
+++
+++SWIGINTERNINLINE PyObject *
+++SWIG_From_float (float value)
+++{
+++ return SWIG_From_double (value);
+++}
+++
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece sentencepiece_ImmutableSentencePieceText__pieces(sentencepiece::ImmutableSentencePieceText const *self,int index){
++ if (index < 0 || index >= static_cast<int>(self->pieces_size())) {
++ throw sentencepiece::util::Status(
++ sentencepiece::util::StatusCode::kOutOfRange,
++@@ -3290,7 +3291,7 @@ SWIGINTERN sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece sent
++ }
++ return self->pieces(index);
++ }
++-SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_ImmutableNBestSentencePieceText_nbests(sentencepiece::ImmutableNBestSentencePieceText const *self,int index){
+++SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_ImmutableNBestSentencePieceText__nbests(sentencepiece::ImmutableNBestSentencePieceText const *self,int index){
++ if (index < 0 || index >= static_cast<int>(self->nbests_size())) {
++ throw sentencepiece::util::Status(
++ sentencepiece::util::StatusCode::kOutOfRange,
++@@ -3590,6 +3591,10 @@ SWIGINTERN BytesArray sentencepiece_SentencePieceProcessor__DecodeIdsAsSerialize
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIdsAsSerializedProto, int,
++ sentencepiece::util::bytes);
++ }
+++SWIGINTERN std::vector< sentencepiece::ImmutableSentencePieceText > sentencepiece_SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< int > > const &ins,int num_threads){
+++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIdsAsImmutableProto, int,
+++ sentencepiece::ImmutableSentencePieceText);
+++ }
++ SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodePiecesBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< absl::string_view > > const &ins,int num_threads){
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodePieces, std::string, std::string);
++ }
++@@ -4070,6 +4075,44 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
+++ int arg2 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText_pieces", 2, 2, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "2"" of type '" "int""'");
+++ }
+++ arg2 = static_cast< int >(val2);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableSentencePieceText const *)arg1)->pieces(arg2);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece(static_cast< const sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_text(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++@@ -4168,7 +4211,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText__pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++ int arg2 ;
++@@ -4179,20 +4222,20 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces(PyObject *SWIGUNUSE
++ PyObject *swig_obj[2] ;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece result;
++
++- if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText_pieces", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText__pieces", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText__pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++ if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "2"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText__pieces" "', argument " "2"" of type '" "int""'");
++ }
++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = sentencepiece_ImmutableSentencePieceText_pieces((sentencepiece::ImmutableSentencePieceText const *)arg1,arg2);
+++ result = sentencepiece_ImmutableSentencePieceText__pieces((sentencepiece::ImmutableSentencePieceText const *)arg1,arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -4299,6 +4342,44 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
+++ int arg2 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val2 ;
+++ int ecode2 = 0 ;
+++ PyObject *swig_obj[2] ;
+++ sentencepiece::ImmutableSentencePieceText result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText_nbests", 2, 2, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
+++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
+++ if (!SWIG_IsOK(ecode2)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "2"" of type '" "int""'");
+++ }
+++ arg2 = static_cast< int >(val2);
+++ {
+++ try {
+++ result = ((sentencepiece::ImmutableNBestSentencePieceText const *)arg1)->nbests(arg2);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText(static_cast< const sentencepiece::ImmutableSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0 );
+++ return resultobj;
+++fail:
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_SerializeAsString(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
++@@ -4332,7 +4413,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText__nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
++ int arg2 ;
++@@ -4343,20 +4424,20 @@ SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests(PyObject *SWIG
++ PyObject *swig_obj[2] ;
++ sentencepiece::ImmutableSentencePieceText result;
++
++- if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText_nbests", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText__nbests", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText__nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++ if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "2"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText__nbests" "', argument " "2"" of type '" "int""'");
++ }
++ arg2 = static_cast< int >(val2);
++ {
++ try {
++- result = sentencepiece_ImmutableNBestSentencePieceText_nbests((sentencepiece::ImmutableNBestSentencePieceText const *)arg1,arg2);
+++ result = sentencepiece_ImmutableNBestSentencePieceText__nbests((sentencepiece::ImmutableNBestSentencePieceText const *)arg1,arg2);
++ ReleaseResultObject(resultobj);
++ }
++ catch (const sentencepiece::util::Status &status) {
++@@ -6822,6 +6903,87 @@ fail:
++ }
++
++
+++SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++ PyObject *resultobj = 0;
+++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
+++ std::vector< std::vector< int > > *arg2 = 0 ;
+++ int arg3 ;
+++ void *argp1 = 0 ;
+++ int res1 = 0 ;
+++ int val3 ;
+++ int ecode3 = 0 ;
+++ PyObject *swig_obj[3] ;
+++ SwigValueWrapper< std::vector< sentencepiece::ImmutableSentencePieceText > > result;
+++
+++ if (!SWIG_Python_UnpackTuple(args, "SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch", 3, 3, swig_obj)) SWIG_fail;
+++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__SentencePieceProcessor, 0 | 0 );
+++ if (!SWIG_IsOK(res1)) {
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch" "', argument " "1"" of type '" "sentencepiece::SentencePieceProcessor const *""'");
+++ }
+++ arg1 = reinterpret_cast< sentencepiece::SentencePieceProcessor * >(argp1);
+++ {
+++ std::vector<std::vector<int>> *out = nullptr;
+++ if (PyList_Check(swig_obj[1])) {
+++ const size_t size = PyList_Size(swig_obj[1]);
+++ out = new std::vector<std::vector<int>>(size);
+++ for (size_t i = 0; i < size; ++i) {
+++ PyObject *o = PyList_GetItem(swig_obj[1], i);
+++ if (PyList_Check(o)) {
+++ const size_t size2 = PyList_Size(o);
+++ (*out)[i].resize(size2);
+++ for (size_t j = 0; j < size2; ++j) {
+++ PyObject *o2 = PyList_GetItem(o, j);
+++ if (PyInt_Check(o2)) {
+++ (*out)[i][j] = static_cast<int>(PyInt_AsLong(o2));
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "list must contain strings");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError, "not a list");
+++ SWIG_fail;
+++ }
+++ }
+++ } else {
+++ PyErr_SetString(PyExc_TypeError,"not a list");
+++ SWIG_fail;
+++ }
+++ arg2 = out;
+++ }
+++ ecode3 = SWIG_AsVal_int(swig_obj[2], &val3);
+++ if (!SWIG_IsOK(ecode3)) {
+++ SWIG_exception_fail(SWIG_ArgError(ecode3), "in method '" "SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch" "', argument " "3"" of type '" "int""'");
+++ }
+++ arg3 = static_cast< int >(val3);
+++ {
+++ try {
+++ result = sentencepiece_SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch((sentencepiece::SentencePieceProcessor const *)arg1,(std::vector< std::vector< int > > const &)*arg2,arg3);
+++ ReleaseResultObject(resultobj);
+++ }
+++ catch (const sentencepiece::util::Status &status) {
+++ SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
+++ }
+++ }
+++ {
+++ resultobj = PyList_New((&result)->size());
+++ for (size_t i = 0; i < (&result)->size(); ++i) {
+++ PyObject *obj = SWIG_NewPointerObj(new sentencepiece::ImmutableSentencePieceText((&result)->at(i)), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0);
+++ PyList_SET_ITEM(resultobj, i, obj);
+++ }
+++ }
+++ {
+++ delete arg2;
+++ }
+++ return resultobj;
+++fail:
+++ {
+++ delete arg2;
+++ }
+++ return NULL;
+++}
+++
+++
++ SWIGINTERN PyObject *_wrap_SentencePieceProcessor__DecodePiecesBatch(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::SentencePieceProcessor *arg1 = (sentencepiece::SentencePieceProcessor *) 0 ;
++@@ -8298,17 +8460,19 @@ static PyMethodDef SwigMethods[] = {
++ { "new_ImmutableSentencePieceText", _wrap_new_ImmutableSentencePieceText, METH_NOARGS, NULL},
++ { "delete_ImmutableSentencePieceText", _wrap_delete_ImmutableSentencePieceText, METH_O, NULL},
++ { "ImmutableSentencePieceText_pieces_size", _wrap_ImmutableSentencePieceText_pieces_size, METH_O, NULL},
+++ { "ImmutableSentencePieceText_pieces", _wrap_ImmutableSentencePieceText_pieces, METH_VARARGS, NULL},
++ { "ImmutableSentencePieceText_text", _wrap_ImmutableSentencePieceText_text, METH_O, NULL},
++ { "ImmutableSentencePieceText_score", _wrap_ImmutableSentencePieceText_score, METH_O, NULL},
++ { "ImmutableSentencePieceText_SerializeAsString", _wrap_ImmutableSentencePieceText_SerializeAsString, METH_O, NULL},
++- { "ImmutableSentencePieceText_pieces", _wrap_ImmutableSentencePieceText_pieces, METH_VARARGS, NULL},
+++ { "ImmutableSentencePieceText__pieces", _wrap_ImmutableSentencePieceText__pieces, METH_VARARGS, NULL},
++ { "ImmutableSentencePieceText_swigregister", ImmutableSentencePieceText_swigregister, METH_O, NULL},
++ { "ImmutableSentencePieceText_swiginit", ImmutableSentencePieceText_swiginit, METH_VARARGS, NULL},
++ { "new_ImmutableNBestSentencePieceText", _wrap_new_ImmutableNBestSentencePieceText, METH_NOARGS, NULL},
++ { "delete_ImmutableNBestSentencePieceText", _wrap_delete_ImmutableNBestSentencePieceText, METH_O, NULL},
++ { "ImmutableNBestSentencePieceText_nbests_size", _wrap_ImmutableNBestSentencePieceText_nbests_size, METH_O, NULL},
++- { "ImmutableNBestSentencePieceText_SerializeAsString", _wrap_ImmutableNBestSentencePieceText_SerializeAsString, METH_O, NULL},
++ { "ImmutableNBestSentencePieceText_nbests", _wrap_ImmutableNBestSentencePieceText_nbests, METH_VARARGS, NULL},
+++ { "ImmutableNBestSentencePieceText_SerializeAsString", _wrap_ImmutableNBestSentencePieceText_SerializeAsString, METH_O, NULL},
+++ { "ImmutableNBestSentencePieceText__nbests", _wrap_ImmutableNBestSentencePieceText__nbests, METH_VARARGS, NULL},
++ { "ImmutableNBestSentencePieceText_swigregister", ImmutableNBestSentencePieceText_swigregister, METH_O, NULL},
++ { "ImmutableNBestSentencePieceText_swiginit", ImmutableNBestSentencePieceText_swiginit, METH_VARARGS, NULL},
++ { "new_SentencePieceProcessor", _wrap_new_SentencePieceProcessor, METH_NOARGS, NULL},
++@@ -8350,6 +8514,7 @@ static PyMethodDef SwigMethods[] = {
++ { "SentencePieceProcessor__DecodePiecesAsImmutableProto", _wrap_SentencePieceProcessor__DecodePiecesAsImmutableProto, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodeIdsBatch", _wrap_SentencePieceProcessor__DecodeIdsBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch", _wrap_SentencePieceProcessor__DecodeIdsAsSerializedProtoBatch, METH_VARARGS, NULL},
+++ { "SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch", _wrap_SentencePieceProcessor__DecodeIdsAsImmutableProtoBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePiecesBatch", _wrap_SentencePieceProcessor__DecodePiecesBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch", _wrap_SentencePieceProcessor__DecodePiecesAsSerializedProtoBatch, METH_VARARGS, NULL},
++ { "SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch", _wrap_SentencePieceProcessor__DecodePiecesAsImmutableProtoBatch, METH_VARARGS, NULL},
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index 2f2c84a..5e4af7f 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -266,6 +266,13 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ t4 = self.sp_.decode_pieces_as_serialized_proto(['foo', 'bar'])
++ t5 = self.sp_.decode_ids_as_serialized_proto([20, 30])
++
+++ y1 = self.sp_.encode(text, out_type='serialized_proto')
+++ y2 = self.sp_.encode(
+++ text, enable_sampling=True, out_type='serialized_proto')
+++ y3 = self.sp_.nbest_encode(text, out_type='serialized_proto', nbest_size=10)
+++ y4 = self.sp_.decode(['foo', 'bar'], out_type='serialized_proto')
+++ y5 = self.sp_.decode([20, 30], out_type='serialized_proto')
+++
++ self.assertEqual(type(s1), bytes)
++ self.assertEqual(type(s2), bytes)
++ self.assertEqual(type(t2), bytes)
++@@ -277,6 +284,92 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(s3, t3)
++ self.assertEqual(s4, t4)
++ self.assertEqual(s5, t5)
+++ self.assertEqual(s1, y1)
+++ self.assertEqual(s3, y3)
+++ self.assertEqual(s4, y4)
+++ self.assertEqual(s5, y5)
+++
+++ ids = self.jasp_.EncodeAsIds(text)
+++ pieces = self.jasp_.EncodeAsPieces(text)
+++ s1 = self.jasp_.EncodeAsSerializedProto(text)
+++ s2 = self.jasp_.DecodeIdsAsSerializedProto(ids)
+++ s3 = self.jasp_.DecodePiecesAsSerializedProto(ids)
+++ self.assertEqual(s2, s1)
+++ self.assertEqual(s3, s1)
+++
+++ def test_immutable_proto(self):
+++ text = 'I saw a girl with a telescope.'
+++ s1 = self.sp_.EncodeAsImmutableProto(text)
+++ s2 = self.sp_.SampleEncodeAsImmutableProto(text, 10, 0.2)
+++ s3 = self.sp_.NBestEncodeAsImmutableProto(text, 10)
+++ s4 = self.sp_.DecodePiecesAsImmutableProto(['foo', 'bar'])
+++ s5 = self.sp_.DecodeIdsAsImmutableProto([20, 30])
+++
+++ t1 = self.sp_.encode_as_immutable_proto(text)
+++ t2 = self.sp_.sample_encode_as_immutable_proto(text, 10, 0.2)
+++ t3 = self.sp_.nbest_encode_as_immutable_proto(text, 10)
+++ t4 = self.sp_.decode_pieces_as_immutable_proto(['foo', 'bar'])
+++ t5 = self.sp_.decode_ids_as_immutable_proto([20, 30])
+++
+++ y1 = self.sp_.encode(text, out_type='immutable_proto')
+++ y2 = self.sp_.encode(text, enable_sampling=True, out_type='immutable_proto')
+++ y3 = self.sp_.nbest_encode(text, out_type='immutable_proto', nbest_size=10)
+++ y4 = self.sp_.decode(['foo', 'bar'], out_type='immutable_proto')
+++ y5 = self.sp_.decode([20, 30], out_type='immutable_proto')
+++
+++ self.assertEqual(s1, t1)
+++ self.assertEqual(s3, t3)
+++ self.assertEqual(s4, t4)
+++ self.assertEqual(s5, t5)
+++ self.assertEqual(s1, y1)
+++ self.assertEqual(s3, y3)
+++ self.assertEqual(s4, y4)
+++ self.assertEqual(s5, y5)
+++
+++ x1 = self.sp_.encode_as_serialized_proto(text)
+++ x2 = self.sp_.sample_encode_as_serialized_proto(text, 10, 0.2)
+++ x3 = self.sp_.nbest_encode_as_serialized_proto(text, 10)
+++ x4 = self.sp_.decode_pieces_as_serialized_proto(['foo', 'bar'])
+++ x5 = self.sp_.decode_ids_as_serialized_proto([20, 30])
+++
+++ self.assertEqual(x1, t1.SerializeAsString())
+++ self.assertEqual(x3, t3.SerializeAsString())
+++ self.assertEqual(x4, t4.SerializeAsString())
+++ self.assertEqual(x5, t5.SerializeAsString())
+++
+++ v1 = self.sp_.EncodeAsIds(text)
+++ v2 = self.sp_.EncodeAsPieces(text)
+++ self.assertEqual([x.id() for x in s1], v1)
+++ self.assertEqual([x.piece() for x in s1], v2)
+++ self.assertEqual(text, s1.text())
+++
+++ surfaces1 = [s1.text()[x.begin():x.end()] for x in s1]
+++ surfaces2 = [x.surface() for x in s1]
+++ self.assertEqual(surfaces1, surfaces2)
+++
+++ ids = []
+++ for i in range(s1.pieces_size()):
+++ ids.append(s1.pieces(i).id())
+++ self.assertEqual(ids, v1)
+++
+++ pieces = []
+++ for i in range(s1.pieces_size()):
+++ pieces.append(s1.pieces(i).piece())
+++ self.assertEqual(pieces, v2)
+++
+++ # Japanese offset
+++ s1 = self.jasp_.EncodeAsImmutableProto('吾輩は猫である。Hello world. ABC 123')
+++ surfaces1 = [s1.text()[x.begin():x.end()] for x in s1]
+++ surfaces2 = [x.surface() for x in s1]
+++ self.assertEqual(surfaces1, surfaces2)
+++
+++ ids = [x.id() for x in s1]
+++ s2 = self.jasp_.DecodeIdsAsImmutableProto(ids)
+++ self.assertEqual(s2, s1)
+++
+++ pieces = [x.piece() for x in s1]
+++ s2 = self.jasp_.DecodePiecesAsImmutableProto(pieces)
+++ self.assertEqual(s2, s1)
++
++ def test_new_api(self):
++ sp = spm.SentencePieceProcessor(
++@@ -386,49 +479,102 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(pieces, sp.encode(text, add_bos=False, add_eos=True))
++
++ def test_sampling(self):
++- sp = spm.SentencePieceProcessor(
++- model_file=os.path.join('test', 'test_model.model'),
++- out_type=str,
++- enable_sampling=True)
++- ids = defaultdict(int)
++- for n in range(100):
++- ++ids[' '.join(sp.encode('hello world'))]
++- self.assertGreater(len(ids), 1)
++-
++- ids2 = defaultdict(int)
++- for n in range(100):
++- ++ids2[' '.join(sp.encode('hello world', enable_sampling=False))]
++- self.assertEqual(len(ids2), 1)
+++ sp = self.sp_
+++
+++ for out_type in [str, int, 'serialized_proto', 'immutable_proto']:
+++ ids = defaultdict(int)
+++ for n in range(100):
+++ out = sp.encode('hello world', out_type=out_type, enable_sampling=True)
+++ if type(out) is list:
+++ out = tuple(out)
+++ ++ids[out]
+++ self.assertGreater(len(ids), 1)
+++
+++ ids2 = defaultdict(int)
+++ for n in range(100):
+++ out = sp.encode('hello world', out_type=out_type, enable_sampling=False)
+++ if type(out) is list:
+++ out = tuple(out)
+++ ++ids2[out]
+++ self.assertEqual(len(ids2), 1)
+++
+++ out = sp.encode(['hello world', 'this is a test'],
+++ out_type=out_type,
+++ enable_sampling=True)
+++ self.assertEqual(len(out), 2)
+++ out = sp.encode(['hello world', 'this is a test'],
+++ out_type=out_type,
+++ enable_sampling=False)
+++ self.assertEqual(len(out), 2)
++
++ def test_nbest(self):
++- sp = spm.SentencePieceProcessor(
++- model_file=os.path.join('test', 'test_model.model'))
+++ sp = self.sp_
++ text = 'hello world'
++- results = sp.nbest_encode(text, nbest_size=10, out_type=str)
++- self.assertEqual(results, sp.NBestEncode(text, nbest_size=10, out_type=str))
++- for n in results:
++- self.assertEqual(sp.decode(n), text)
++- decoded = sp.decode(results)
++- for n in decoded:
++- self.assertEqual(n, text)
++- results = sp.nbest_encode(text, nbest_size=10, out_type=int)
++- self.assertEqual(results, sp.NBestEncode(text, nbest_size=10, out_type=int))
++- for n in results:
++- self.assertEqual(sp.decode(n), text)
++- decoded = sp.decode(results)
++- for n in decoded:
++- self.assertEqual(n, text)
+++ text2 = 'I have a pen.'
+++
+++ for out_type in [str, int, 'serialized_proto', 'immutable_proto']:
+++ results = sp.nbest_encode(text, nbest_size=10, out_type=out_type)
+++ self.assertEqual(results,
+++ sp.NBestEncode(text, nbest_size=10, out_type=out_type))
+++
+++ if out_type in [str, int]:
+++ for n in results:
+++ self.assertEqual(sp.decode(n), text)
+++
+++ for n in sp.decode(results):
+++ self.assertEqual(n, text)
+++
+++ # batch test
+++ results = sp.nbest_encode([text, text2], nbest_size=10, out_type=out_type)
+++ self.assertEqual(
+++ results,
+++ sp.NBestEncode([text, text2], nbest_size=10, out_type=out_type))
+++ self.assertEqual(len(results), 2)
+++
+++ if out_type in [str, int]:
+++ for n in results[0]:
+++ self.assertEqual(sp.decode(n), text)
+++
+++ for n in results[1]:
+++ self.assertEqual(sp.decode(n), text2)
+++
+++ decoded = sp.decode(results[0])
+++ self.assertEqual(len(decoded), 10)
+++ for n in decoded:
+++ self.assertEqual(n, text)
+++ decoded = sp.decode(results[1])
+++ self.assertEqual(len(decoded), 10)
+++ for n in decoded:
+++ self.assertEqual(n, text2)
++
++ def test_sample_and_score(self):
++- sp = spm.SentencePieceProcessor(
++- model_file=os.path.join('test', 'test_model.model'))
+++ sp = self.sp_
++ text = 'hello world'
++- results = sp.sample_encode_and_score(text, wor=True, out_type=str)
++- for n in results:
++- self.assertEqual(sp.decode(n[0]), text)
++- results = sp.sample_encode_and_score(text, wor=True, out_type=int)
++- for n in results:
++- self.assertEqual(sp.decode(n[0]), text)
+++ text2 = 'I have a pen.'
+++ for out_type in [str, int, 'serialized_proto', 'immutable_proto']:
+++ results = sp.sample_encode_and_score(
+++ text, wor=True, num_samples=10, out_type=out_type)
+++ results = sp.SampleEncodeAndScore(
+++ text, wor=False, num_samples=10, out_type=out_type)
+++
+++ if out_type in [str, int]:
+++ for n in results:
+++ self.assertEqual(sp.decode(n[0]), text)
+++
+++ results = sp.sample_encode_and_score([text, text2],
+++ wor=True,
+++ num_samples=10,
+++ out_type=out_type)
+++ results = sp.SampleEncodeAndScore([text, text2],
+++ wor=True,
+++ num_samples=10,
+++ out_type=out_type)
+++
+++ if out_type in [str, int]:
+++ for n in results[0]:
+++ self.assertEqual(sp.decode(n[0]), text)
+++ for n in results[1]:
+++ self.assertEqual(sp.decode(n[0]), text2)
++
++ def test_valid_range(self):
++ size = self.sp_.piece_size()
++@@ -452,65 +598,28 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ with open(os.path.join(data_dir, 'botchan.txt'), 'r') as file:
++ texts = file.readlines()
++
++- r1 = sp.encode(texts, out_type=str, num_threads=None)
++- r2 = sp.encode(texts, out_type=str, num_threads=1)
++- r3 = sp.encode(texts, out_type=str, num_threads=-1)
++- r4 = sp.encode(texts, out_type=str, num_threads=8)
++- r5 = [sp.encode(s, out_type=str) for s in texts]
++- self.assertEqual(r1, r2)
++- self.assertEqual(r1, r3)
++- self.assertEqual(r1, r4)
++- self.assertEqual(r1, r5)
++-
++- d1 = sp.decode(r1, num_threads=None)
++- d2 = sp.decode(r2, num_threads=1)
++- d3 = sp.decode(r3, num_threads=-1)
++- d4 = sp.decode(r4, num_threads=8)
++- d5 = [sp.decode(s) for s in r5]
++- self.assertEqual(d1, d2)
++- self.assertEqual(d1, d3)
++- self.assertEqual(d1, d4)
++- self.assertEqual(d1, d5)
++-
++- r1 = sp.encode(texts, out_type=int, num_threads=None)
++- r2 = sp.encode(texts, out_type=int, num_threads=1)
++- r3 = sp.encode(texts, out_type=int, num_threads=-1)
++- r4 = sp.encode(texts, out_type=int, num_threads=8)
++- r5 = [sp.encode(s, out_type=int) for s in texts]
++- self.assertEqual(r1, r2)
++- self.assertEqual(r1, r3)
++- self.assertEqual(r1, r4)
++- self.assertEqual(r1, r5)
++-
++- d1 = sp.decode(r1, num_threads=None)
++- d2 = sp.decode(r2, num_threads=1)
++- d3 = sp.decode(r3, num_threads=-1)
++- d4 = sp.decode(r4, num_threads=8)
++- d5 = [sp.decode(s) for s in r5]
++- self.assertEqual(d1, d2)
++- self.assertEqual(d1, d3)
++- self.assertEqual(d1, d4)
++- self.assertEqual(d1, d5)
++-
++- r1 = sp.encode(texts, out_type='serialized_proto', num_threads=None)
++- r2 = sp.encode(texts, out_type='serialized_proto', num_threads=1)
++- r3 = sp.encode(texts, out_type='serialized_proto', num_threads=-1)
++- r4 = sp.encode(texts, out_type='serialized_proto', num_threads=8)
++- r5 = [sp.encode(s, out_type='serialized_proto') for s in texts]
++- self.assertEqual(r1, r2)
++- self.assertEqual(r1, r3)
++- self.assertEqual(r1, r4)
++- self.assertEqual(r1, r5)
++-
++- r1 = sp.encode(texts, out_type='immutable_proto', num_threads=None)
++- r2 = sp.encode(texts, out_type='immutable_proto', num_threads=1)
++- r3 = sp.encode(texts, out_type='immutable_proto', num_threads=-1)
++- r4 = sp.encode(texts, out_type='immutable_proto', num_threads=8)
++- r5 = [sp.encode(s, out_type='immutable_proto') for s in texts]
++- self.assertEqual(r1, r2)
++- self.assertEqual(r1, r3)
++- self.assertEqual(r1, r4)
++- self.assertEqual(r1, r5)
+++ for out_type in [str, int, 'serialized_proto', 'immutable_proto']:
+++ r1 = sp.encode(texts, out_type=out_type, num_threads=None)
+++ r2 = sp.encode(texts, out_type=out_type, num_threads=1)
+++ r3 = sp.encode(texts, out_type=out_type, num_threads=-1)
+++ r4 = sp.encode(texts, out_type=out_type, num_threads=8)
+++ r5 = [sp.encode(s, out_type=out_type) for s in texts]
+++ self.assertEqual(r1, r2)
+++ self.assertEqual(r1, r3)
+++ self.assertEqual(r1, r4)
+++ self.assertEqual(r1, r5)
+++
+++ if out_type in [str, int]:
+++ d1 = sp.decode(r1, num_threads=None)
+++ d2 = sp.decode(r2, num_threads=1)
+++ d3 = sp.decode(r3, num_threads=-1)
+++ d4 = sp.decode(r4, num_threads=8)
+++ d5 = [sp.decode(s) for s in r5]
+++
+++ self.assertEqual(d1, d2)
+++ self.assertEqual(d1, d3)
+++ self.assertEqual(d1, d4)
+++ self.assertEqual(d1, d5)
++
++ e1 = sp.calculate_entropy(texts, alpha=1.0, num_threads=10)
++ e2 = sp.CalculateEntropy(texts, alpha=1.0, num_threads=10)
++diff --git a/src/sentencepiece_processor.cc b/src/sentencepiece_processor.cc
++index 482a45b..2a5c399 100644
++--- a/src/sentencepiece_processor.cc
+++++ b/src/sentencepiece_processor.cc
++@@ -55,6 +55,34 @@ std::vector<absl::string_view> ToPieceArray(const std::vector<std::string> &v) {
++ return out;
++ }
++
+++void ConvertToUnicodeSpansInternal(SentencePieceText *spt) {
+++ if (spt == nullptr) return;
+++
+++ std::vector<int> utf8_to_unicode(spt->text().size() + 1, 0);
+++ absl::string_view str = spt->text();
+++ size_t prev = 0;
+++ int ulen = 0;
+++ while (!str.empty()) {
+++ const size_t mblen = string_util::OneCharLen(str.data());
+++ for (int i = prev; i < prev + mblen; ++i) {
+++ utf8_to_unicode[i] = ulen;
+++ }
+++ ++ulen;
+++ prev += mblen;
+++ str.remove_prefix(mblen);
+++ }
+++ utf8_to_unicode[prev] = ulen;
+++
+++ auto clip = [&](int s) {
+++ return std::min<int>(std::max<int>(0, s), utf8_to_unicode.size() - 1);
+++ };
+++
+++ for (auto &piece : *(spt->mutable_pieces())) {
+++ piece.set_begin(utf8_to_unicode[clip(piece.begin())]);
+++ piece.set_end(utf8_to_unicode[clip(piece.end())]);
+++ }
+++}
+++
++ } // namespace
++
++ ImmutableSentencePieceText::ImmutableSentencePieceText()
++@@ -132,6 +160,10 @@ SentencePieceText *ImmutableSentencePieceText::mutable_proto() {
++ return rep_.get();
++ }
++
+++void ImmutableSentencePieceText::ConvertToUnicodeSpans() {
+++ ConvertToUnicodeSpansInternal(mutable_proto());
+++}
+++
++ util::bytes ImmutableSentencePieceText::SerializeAsString() const {
++ return spt_->SerializeAsString();
++ }
++@@ -164,6 +196,13 @@ NBestSentencePieceText *ImmutableNBestSentencePieceText::mutable_proto() {
++ return rep_.get();
++ }
++
+++void ImmutableNBestSentencePieceText::ConvertToUnicodeSpans() {
+++ if (!mutable_proto()) return;
+++ for (auto &spt : *(mutable_proto()->mutable_nbests())) {
+++ ConvertToUnicodeSpansInternal(&spt);
+++ }
+++}
+++
++ util::bytes ImmutableNBestSentencePieceText::SerializeAsString() const {
++ return rep_ ? rep_->SerializeAsString() : "";
++ }
++@@ -1048,34 +1087,6 @@ std::string SentencePieceProcessor::serialized_model_proto() const {
++ // std::random_device.
++ void SetRandomGeneratorSeed(unsigned int seed);
++
++-void ConvertToUnicodeSpans(SentencePieceText *spt) {
++- if (spt == nullptr) return;
++-
++- std::vector<int> utf8_to_unicode(spt->text().size() + 1, 0);
++- absl::string_view str = spt->text();
++- size_t prev = 0;
++- int ulen = 0;
++- while (!str.empty()) {
++- const size_t mblen = string_util::OneCharLen(str.data());
++- for (int i = prev; i < prev + mblen; ++i) {
++- utf8_to_unicode[i] = ulen;
++- }
++- ++ulen;
++- prev += mblen;
++- str.remove_prefix(mblen);
++- }
++- utf8_to_unicode[prev] = ulen;
++-
++- auto clip = [&](int s) {
++- return std::min<int>(std::max<int>(0, s), utf8_to_unicode.size() - 1);
++- };
++-
++- for (auto &piece : *(spt->mutable_pieces())) {
++- piece.set_begin(utf8_to_unicode[clip(piece.begin())]);
++- piece.set_end(utf8_to_unicode[clip(piece.end())]);
++- }
++-}
++-
++ namespace io {
++ util::Status LoadModelProto(absl::string_view filename,
++ ModelProto *model_proto) {
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index b7fae6a..d107a2a 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -25,8 +25,8 @@
++ #ifndef SWIG
++ namespace absl {
++ using std::string_view;
++-}
++-#endif // SWIG
+++} // namespace absl
+++#endif
++
++ namespace sentencepiece {
++ namespace util {
++@@ -196,6 +196,9 @@ class ImmutableSentencePieceText {
++ // it returns the raw pointer managed by the shared_ptr.
++ SentencePieceText *mutable_proto();
++
+++ // Converts the utf8 byte spans into Unicode char span.
+++ void ConvertToUnicodeSpans();
+++
++ friend class ImmutableNBestSentencePieceText;
++
++ private:
++@@ -225,6 +228,8 @@ class ImmutableNBestSentencePieceText {
++ // it returns the raw pointer managed by the shared_ptr.
++ NBestSentencePieceText *mutable_proto();
++
+++ void ConvertToUnicodeSpans();
+++
++ private:
++ std::shared_ptr<NBestSentencePieceText> rep_;
++ };
++@@ -415,14 +420,16 @@ class SentencePieceProcessor {
++ virtual util::Status Decode(const std::vector<int> &ids,
++ SentencePieceText *spt) const;
++
++-#ifdef SWIG
+++#ifdef SWIGPYTHON
+++#define CONVERT_TO_UNICODE_SPAN output.ConvertToUnicodeSpans();
++ #define SPP_SWIG_CHECK_AND_THROW \
++ if (!status.ok()) throw status;
++ #else
+++#define CONVERT_TO_UNICODE_SPAN
++ #define SPP_SWIG_CHECK_AND_THROW \
++ if (!status.ok()) { \
++ }
++-#endif // SWIG
+++#endif // SWIGPYTHON
++
++ #define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++@@ -439,6 +446,7 @@ class SentencePieceProcessor {
++ #define DEFINE_SPP_IMMUTABLE_PROTO_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
+++ CONVERT_TO_UNICODE_SPAN; \
++ SPP_SWIG_CHECK_AND_THROW; \
++ return output;
++
++@@ -707,9 +715,6 @@ class SentencePieceProcessor {
++ // std::random_device.
++ void SetRandomGeneratorSeed(unsigned int seed);
++
++-// Converts the utf8 byte spans into Unicode char span.
++-void ConvertToUnicodeSpans(SentencePieceText *spt);
++-
++ #ifndef SWIG
++ // IO related functions to absorb model formats.
++ namespace io {
++diff --git a/src/sentencepiece_processor_test.cc b/src/sentencepiece_processor_test.cc
++index ff55aeb..f05dc5d 100644
++--- a/src/sentencepiece_processor_test.cc
+++++ b/src/sentencepiece_processor_test.cc
++@@ -1657,11 +1657,12 @@ TEST(SentencePieceProcessorTest, ImmutableNBestSentencePieceTextTest) {
++
++ TEST(SentencePieceProcessorTest, ConvertToUnicodeSpansTest) {
++ auto make_spt = [&](const std::vector<std::string> &tokens) {
++- SentencePieceText spt;
+++ ImmutableSentencePieceText ispt;
+++ auto *spt = ispt.mutable_proto();
++ int prev = 0;
++ std::string text;
++ for (const auto &tok : tokens) {
++- auto *piece = spt.add_pieces();
+++ auto *piece = spt->add_pieces();
++ piece->set_surface(tok);
++ piece->set_piece(tok);
++ piece->set_begin(prev);
++@@ -1669,9 +1670,9 @@ TEST(SentencePieceProcessorTest, ConvertToUnicodeSpansTest) {
++ prev += tok.size();
++ text += tok;
++ }
++- spt.set_text(text);
++- ConvertToUnicodeSpans(&spt);
++- return spt;
+++ spt->set_text(text);
+++ ispt.ConvertToUnicodeSpans();
+++ return ispt;
++ };
++
++ {
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 3 Aug 2022 12:45:31 +0900
++Subject: Adds SWIGPYTHON flag
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/setup.py | 3 ++-
++ python/src/sentencepiece/__init__.py | 2 +-
++ 2 files changed, 3 insertions(+), 2 deletions(-)
++
++diff --git a/python/setup.py b/python/setup.py
++index fdf9394..3438ddd 100755
++--- a/python/setup.py
+++++ b/python/setup.py
++@@ -96,6 +96,7 @@ class build_ext(_build_ext):
++ else:
++ cflags.append('-Wl,-strip-all')
++ libs.append('-Wl,-strip-all')
+++ cflags.append('-DSWIGPYTHON')
++ print('## cflags={}'.format(' '.join(cflags)))
++ print('## libs={}'.format(' '.join(libs)))
++ ext.extra_compile_args = cflags
++@@ -115,7 +116,7 @@ if os.name == 'nt':
++ '..\\build\\root_{}\\lib\\sentencepiece_train.lib'.format(arch)
++ ]
++ else:
++- cflags = ['/std:c++17', '/MT', '/I..\\build\\root\\include']
+++ cflags = ['/std:c++17', '/MT', '/I..\\build\\root\\include', '/DSWIGPYTHON']
++ libs = [
++ '..\\build\\root\\lib\\sentencepiece.lib',
++ '..\\build\\root\\lib\\sentencepiece_train.lib'
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index 07acb94..2a91022 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -126,7 +126,7 @@ class ImmutableSentencePieceText(object):
++ return self.SerializeAsString() == other.SerializeAsString()
++
++ def __hash__(self):
++- return hash(self.SerializeAsString())
+++ return hash(self.SerializeAsString())
++
++
++ # Register ImmutableSentencePieceText in _sentencepiece:
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 3 Aug 2022 15:45:09 +0900
++Subject: remove unused ifdef SWIG macro
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/sentencepiece.i | 5 ++++
++ src/sentencepiece_processor.h | 42 ++++++++++++++++++--------------
++ 2 files changed, 29 insertions(+), 18 deletions(-)
++
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index f3a4f30..75f62c8 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -326,6 +326,8 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ %ignore sentencepiece::SentencePieceProcessor::model_proto;
++ %ignore sentencepiece::SentencePieceProcessor::Load;
++ %ignore sentencepiece::SentencePieceProcessor::LoadOrDie;
+++%ignore sentencepiece::SentencePieceProcessor::SetModel;
+++%ignore sentencepiece::SentencePieceProcessor::SetNormalizer;
++ %ignore sentencepiece::pretokenizer::PretokenizerForTrainingInterface;
++ %ignore sentencepiece::SentenceIterator;
++ %ignore sentencepiece::ConvertToUnicodeSpans;
++@@ -339,6 +341,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ %ignore sentencepiece::SentencePieceTrainer::SetPretokenizerForTraining;
++ %ignore sentencepiece::SentencePieceTrainer::GetPretokenizerForTraining;
++
+++%ignore sentencepiece::io::LoadModelProto;
+++%ignore sentencepiece::io::SaveModelProto;
+++
++ %extend sentencepiece::SentencePieceProcessor {
++ sentencepiece::util::Status LoadFromFile(absl::string_view arg) {
++ return $self->Load(arg);
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index d107a2a..be9449e 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -26,7 +26,7 @@
++ namespace absl {
++ using std::string_view;
++ } // namespace absl
++-#endif
+++#endif // SWIG
++
++ namespace sentencepiece {
++ namespace util {
++@@ -420,36 +420,46 @@ class SentencePieceProcessor {
++ virtual util::Status Decode(const std::vector<int> &ids,
++ SentencePieceText *spt) const;
++
++-#ifdef SWIGPYTHON
++-#define CONVERT_TO_UNICODE_SPAN output.ConvertToUnicodeSpans();
++-#define SPP_SWIG_CHECK_AND_THROW \
++- if (!status.ok()) throw status;
+++#ifndef SWIGPYTHON
+++
+++#define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
+++ OutType output; \
+++ const auto status = FuncName(__VA_ARGS__, &output); \
+++ return output;
+++
+++#define DEFINE_SPP_SERIALIZED_PROTO_IMPL(FuncName, OutType, ...) \
+++ OutType output; \
+++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
+++ return output.SerializeAsString();
+++
+++#define DEFINE_SPP_IMMUTABLE_PROTO_IMPL(FuncName, OutType, ...) \
+++ OutType output; \
+++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
+++ return output;
+++
++ #else
++-#define CONVERT_TO_UNICODE_SPAN
++-#define SPP_SWIG_CHECK_AND_THROW \
++- if (!status.ok()) { \
++- }
++-#endif // SWIGPYTHON
++
++ #define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, &output); \
++- SPP_SWIG_CHECK_AND_THROW; \
+++ if (!status.ok()) throw status; \
++ return output;
++
++ #define DEFINE_SPP_SERIALIZED_PROTO_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
++- SPP_SWIG_CHECK_AND_THROW; \
+++ if (!status.ok()) throw status; \
++ return output.SerializeAsString();
++
++ #define DEFINE_SPP_IMMUTABLE_PROTO_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
++- CONVERT_TO_UNICODE_SPAN; \
++- SPP_SWIG_CHECK_AND_THROW; \
+++ if (!status.ok()) throw status; \
+++ output.ConvertToUnicodeSpans(); \
++ return output;
++
+++#endif // SWIGPYTHON
+++
++ //////////////////////////////////////////////////////////////
++ // Handy methods that return the result directly.
++ // These functions ignore internal errors.
++@@ -664,7 +674,6 @@ class SentencePieceProcessor {
++ // Returns PAD (<pad>) id.
++ virtual int pad_id() const;
++
++-#ifndef SWIG
++ //////////////////////////////////////////////////////////////
++ // Model management.
++ //
++@@ -673,7 +682,6 @@ class SentencePieceProcessor {
++
++ // Allows injection of a normalizer instance. `normalizer` is moved.
++ void SetNormalizer(std::unique_ptr<normalizer::Normalizer> &&normalizer);
++-#endif // SWIG
++
++ // Returns immutable model proto. Useful to obtain extended
++ // or experimental parameters encoded in model_proto.
++@@ -715,7 +723,6 @@ class SentencePieceProcessor {
++ // std::random_device.
++ void SetRandomGeneratorSeed(unsigned int seed);
++
++-#ifndef SWIG
++ // IO related functions to absorb model formats.
++ namespace io {
++ // Loads `model_proto` from `filename`.
++@@ -730,6 +737,5 @@ util::Status LoadModelProto(absl::string_view, ModelProto *model_proto);
++ // Saves `model_proto` as `filename`.
++ util::Status SaveModelProto(absl::string_view, const ModelProto &model_proto);
++ } // namespace io
++-#endif // SWIG
++ } // namespace sentencepiece
++ #endif // SENTENCEPIECE_PROCESSOR_H_
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Wed, 3 Aug 2022 17:20:01 +0900
++Subject: Fixed test failure.
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/sentencepiece.i | 35 +++++++++++++++++++++----
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 35 +++++++++++++++++++++----
++ src/sentencepiece_processor.cc | 4 +--
++ src/sentencepiece_processor.h | 34 +++++++-----------------
++ 4 files changed, 72 insertions(+), 36 deletions(-)
++
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 75f62c8..1a94fef 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -193,6 +193,19 @@ inline void CheckIds(const std::vector<int> &ids, int num_pieces) {
++
++ inline void CheckIds(const std::vector<absl::string_view> &ids, int num_pieces) {}
++
+++template <typename T>
+++inline void ConvertToUnicodeSpans(T *proto) {}
+++
+++template <>
+++inline void ConvertToUnicodeSpans(sentencepiece::ImmutableSentencePieceText *proto) {
+++ proto->ConvertToUnicodeSpans();
+++}
+++
+++template <>
+++inline void ConvertToUnicodeSpans(sentencepiece::ImmutableNBestSentencePieceText *proto) {
+++ proto->ConvertToUnicodeSpans();
+++}
+++
++ class ThreadPool {
++ public:
++ explicit ThreadPool(size_t request_size) :
++@@ -239,6 +252,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ self->FuncName(ins[i]); \
++ RewriteIds(*self, &out, add_bos, add_eos, reverse, \
++ emit_unk_piece); \
+++ ConvertToUnicodeSpans(&out); \
++ outs[i] = std::move(out); \
++ } \
++ }); \
++@@ -255,7 +269,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ pool.Schedule([&, n]() { \
++ for (size_t i = n; i < ins.size(); i += num_threads) { \
++ CheckIds(ins[i], self->GetPieceSize()); \
++- outs[i] = self->FuncName(ins[i]); \
+++ auto out = self->FuncName(ins[i]); \
+++ ConvertToUnicodeSpans(&out); \
+++ outs[i] = std::move(out); \
++ } \
++ }); \
++ } \
++@@ -396,6 +412,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ auto proto = enable_sampling ?
++ $self->SampleEncodeAsImmutableProto(text, nbest_size, alpha) :
++ $self->EncodeAsImmutableProto(text);
+++ proto.ConvertToUnicodeSpans();
++ RewriteIds(*$self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
++ return proto;
++ }
++@@ -467,13 +484,17 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ sentencepiece::ImmutableSentencePieceText _DecodeIdsAsImmutableProto(
++ const std::vector<int> &ids) const {
++ CheckIds(ids, $self->GetPieceSize());
++- return $self->DecodeIdsAsImmutableProto(ids);
+++ auto proto = $self->DecodeIdsAsImmutableProto(ids);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++
++ sentencepiece::ImmutableSentencePieceText _DecodePiecesAsImmutableProto(
++ const std::vector<absl::string_view> &pieces) const {
++ CheckIds(pieces, $self->GetPieceSize());
++- return $self->DecodePiecesAsImmutableProto(pieces);
+++ auto proto= $self->DecodePiecesAsImmutableProto(pieces);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++
++ /////////////////////////////////////////////////////////////////////////////
++@@ -557,7 +578,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ bool emit_unk_piece) const {
++ RewriteIds(*$self, static_cast<sentencepiece::ImmutableSentencePieceText *>(nullptr),
++ add_bos, add_eos, reverse, emit_unk_piece);
++- return $self->NBestEncodeAsImmutableProto(text, nbest_size);
+++ auto proto = $self->NBestEncodeAsImmutableProto(text, nbest_size);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++
++
++@@ -611,8 +634,10 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ bool emit_unk_piece) const {
++ RewriteIds(*$self, static_cast<sentencepiece::util::bytes *>(nullptr),
++ add_bos, add_eos, reverse, emit_unk_piece);
++- return $self->SampleEncodeAndScoreAsImmutableProto(text, num_samples,
+++ auto proto = $self->SampleEncodeAndScoreAsImmutableProto(text, num_samples,
++ alpha, wor, include_best);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++
++
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 22e0708..4b8b5ef 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -3002,6 +3002,19 @@ inline void CheckIds(const std::vector<int> &ids, int num_pieces) {
++
++ inline void CheckIds(const std::vector<absl::string_view> &ids, int num_pieces) {}
++
+++template <typename T>
+++inline void ConvertToUnicodeSpans(T *proto) {}
+++
+++template <>
+++inline void ConvertToUnicodeSpans(sentencepiece::ImmutableSentencePieceText *proto) {
+++ proto->ConvertToUnicodeSpans();
+++}
+++
+++template <>
+++inline void ConvertToUnicodeSpans(sentencepiece::ImmutableNBestSentencePieceText *proto) {
+++ proto->ConvertToUnicodeSpans();
+++}
+++
++ class ThreadPool {
++ public:
++ explicit ThreadPool(size_t request_size) :
++@@ -3048,6 +3061,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ self->FuncName(ins[i]); \
++ RewriteIds(*self, &out, add_bos, add_eos, reverse, \
++ emit_unk_piece); \
+++ ConvertToUnicodeSpans(&out); \
++ outs[i] = std::move(out); \
++ } \
++ }); \
++@@ -3064,7 +3078,9 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ pool.Schedule([&, n]() { \
++ for (size_t i = n; i < ins.size(); i += num_threads) { \
++ CheckIds(ins[i], self->GetPieceSize()); \
++- outs[i] = self->FuncName(ins[i]); \
+++ auto out = self->FuncName(ins[i]); \
+++ ConvertToUnicodeSpans(&out); \
+++ outs[i] = std::move(out); \
++ } \
++ }); \
++ } \
++@@ -3540,6 +3556,7 @@ SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_SentencePiece
++ auto proto = enable_sampling ?
++ self->SampleEncodeAsImmutableProto(text, nbest_size, alpha) :
++ self->EncodeAsImmutableProto(text);
+++ proto.ConvertToUnicodeSpans();
++ RewriteIds(*self, &proto, add_bos, add_eos, reverse, emit_unk_piece);
++ return proto;
++ }
++@@ -3578,11 +3595,15 @@ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__Deco
++ }
++ SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_SentencePieceProcessor__DecodeIdsAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,std::vector< int > const &ids){
++ CheckIds(ids, self->GetPieceSize());
++- return self->DecodeIdsAsImmutableProto(ids);
+++ auto proto = self->DecodeIdsAsImmutableProto(ids);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++ SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_SentencePieceProcessor__DecodePiecesAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,std::vector< absl::string_view > const &pieces){
++ CheckIds(pieces, self->GetPieceSize());
++- return self->DecodePiecesAsImmutableProto(pieces);
+++ auto proto= self->DecodePiecesAsImmutableProto(pieces);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++ SWIGINTERN std::vector< std::string > sentencepiece_SentencePieceProcessor__DecodeIdsBatch(sentencepiece::SentencePieceProcessor const *self,std::vector< std::vector< int > > const &ins,int num_threads){
++ DEFINE_DECODE_BATCH_FUNC_IMPL(DecodeIds, int, std::string);
++@@ -3628,7 +3649,9 @@ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__NBes
++ SWIGINTERN sentencepiece::ImmutableNBestSentencePieceText sentencepiece_SentencePieceProcessor__NBestEncodeAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int nbest_size,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ RewriteIds(*self, static_cast<sentencepiece::ImmutableSentencePieceText *>(nullptr),
++ add_bos, add_eos, reverse, emit_unk_piece);
++- return self->NBestEncodeAsImmutableProto(text, nbest_size);
+++ auto proto = self->NBestEncodeAsImmutableProto(text, nbest_size);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++ SWIGINTERN std::vector< std::pair< std::vector< int >,float > > sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsIds(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float alpha,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ auto idss = self->SampleEncodeAndScoreAsIds(text, num_samples,
++@@ -3655,8 +3678,10 @@ SWIGINTERN sentencepiece::util::bytes sentencepiece_SentencePieceProcessor__Samp
++ SWIGINTERN sentencepiece::ImmutableNBestSentencePieceText sentencepiece_SentencePieceProcessor__SampleEncodeAndScoreAsImmutableProto(sentencepiece::SentencePieceProcessor const *self,absl::string_view text,int num_samples,float alpha,bool wor,bool include_best,bool add_bos,bool add_eos,bool reverse,bool emit_unk_piece){
++ RewriteIds(*self, static_cast<sentencepiece::util::bytes *>(nullptr),
++ add_bos, add_eos, reverse, emit_unk_piece);
++- return self->SampleEncodeAndScoreAsImmutableProto(text, num_samples,
+++ auto proto = self->SampleEncodeAndScoreAsImmutableProto(text, num_samples,
++ alpha, wor, include_best);
+++ proto.ConvertToUnicodeSpans();
+++ return proto;
++ }
++ SWIGINTERN float sentencepiece_SentencePieceProcessor__CalculateEntropy(sentencepiece::SentencePieceProcessor *self,absl::string_view text,float alpha){
++ return self->CalculateEntropy(text, alpha);
++diff --git a/src/sentencepiece_processor.cc b/src/sentencepiece_processor.cc
++index 2a5c399..f0df2f6 100644
++--- a/src/sentencepiece_processor.cc
+++++ b/src/sentencepiece_processor.cc
++@@ -56,14 +56,14 @@ std::vector<absl::string_view> ToPieceArray(const std::vector<std::string> &v) {
++ }
++
++ void ConvertToUnicodeSpansInternal(SentencePieceText *spt) {
++- if (spt == nullptr) return;
+++ if (spt == nullptr || spt->text().empty()) return;
++
++ std::vector<int> utf8_to_unicode(spt->text().size() + 1, 0);
++ absl::string_view str = spt->text();
++ size_t prev = 0;
++ int ulen = 0;
++ while (!str.empty()) {
++- const size_t mblen = string_util::OneCharLen(str.data());
+++ const size_t mblen = std::max<int>(1, string_util::OneCharLen(str.data()));
++ for (int i = prev; i < prev + mblen; ++i) {
++ utf8_to_unicode[i] = ulen;
++ }
++diff --git a/src/sentencepiece_processor.h b/src/sentencepiece_processor.h
++index be9449e..14b1e8c 100644
++--- a/src/sentencepiece_processor.h
+++++ b/src/sentencepiece_processor.h
++@@ -419,47 +419,33 @@ class SentencePieceProcessor {
++
++ virtual util::Status Decode(const std::vector<int> &ids,
++ SentencePieceText *spt) const;
++-
++-#ifndef SWIGPYTHON
++-
++-#define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
++- OutType output; \
++- const auto status = FuncName(__VA_ARGS__, &output); \
++- return output;
++-
++-#define DEFINE_SPP_SERIALIZED_PROTO_IMPL(FuncName, OutType, ...) \
++- OutType output; \
++- const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
++- return output.SerializeAsString();
++-
++-#define DEFINE_SPP_IMMUTABLE_PROTO_IMPL(FuncName, OutType, ...) \
++- OutType output; \
++- const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
++- return output;
++-
+++#ifdef SWIG
+++#define SPP_SWIG_CHECK_AND_THROW \
+++ if (!status.ok()) throw status;
++ #else
+++#define SPP_SWIG_CHECK_AND_THROW \
+++ if (!status.ok()) { \
+++ }
+++#endif // SWIG
++
++ #define DEFINE_SPP_DIRECT_FUNC_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, &output); \
++- if (!status.ok()) throw status; \
+++ SPP_SWIG_CHECK_AND_THROW; \
++ return output;
++
++ #define DEFINE_SPP_SERIALIZED_PROTO_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
++- if (!status.ok()) throw status; \
+++ SPP_SWIG_CHECK_AND_THROW; \
++ return output.SerializeAsString();
++
++ #define DEFINE_SPP_IMMUTABLE_PROTO_IMPL(FuncName, OutType, ...) \
++ OutType output; \
++ const auto status = FuncName(__VA_ARGS__, output.mutable_proto()); \
++- if (!status.ok()) throw status; \
++- output.ConvertToUnicodeSpans(); \
+++ SPP_SWIG_CHECK_AND_THROW; \
++ return output;
++
++-#endif // SWIGPYTHON
++-
++ //////////////////////////////////////////////////////////////
++ // Handy methods that return the result directly.
++ // These functions ignore internal errors.
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Thu, 4 Aug 2022 16:03:31 +0900
++Subject: Uses property in immutable proto
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/setup.py | 3 +-
++ python/src/sentencepiece/__init__.py | 128 ++++++++++++------
++ python/src/sentencepiece/sentencepiece.i | 143 ++++++++++++++------
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 168 ++++++------------------
++ python/test/sentencepiece_test.py | 68 +++++-----
++ 5 files changed, 265 insertions(+), 245 deletions(-)
++
++diff --git a/python/setup.py b/python/setup.py
++index 3438ddd..fdf9394 100755
++--- a/python/setup.py
+++++ b/python/setup.py
++@@ -96,7 +96,6 @@ class build_ext(_build_ext):
++ else:
++ cflags.append('-Wl,-strip-all')
++ libs.append('-Wl,-strip-all')
++- cflags.append('-DSWIGPYTHON')
++ print('## cflags={}'.format(' '.join(cflags)))
++ print('## libs={}'.format(' '.join(libs)))
++ ext.extra_compile_args = cflags
++@@ -116,7 +115,7 @@ if os.name == 'nt':
++ '..\\build\\root_{}\\lib\\sentencepiece_train.lib'.format(arch)
++ ]
++ else:
++- cflags = ['/std:c++17', '/MT', '/I..\\build\\root\\include', '/DSWIGPYTHON']
+++ cflags = ['/std:c++17', '/MT', '/I..\\build\\root\\include']
++ libs = [
++ '..\\build\\root\\lib\\sentencepiece.lib',
++ '..\\build\\root\\lib\\sentencepiece_train.lib'
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index 2a91022..12dc631 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -69,20 +69,36 @@ class ImmutableSentencePieceText_ImmutableSentencePiece(object):
++ _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_swiginit(self, _sentencepiece.new_ImmutableSentencePieceText_ImmutableSentencePiece())
++ __swig_destroy__ = _sentencepiece.delete_ImmutableSentencePieceText_ImmutableSentencePiece
++
++- def piece(self):
++- return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_piece(self)
+++ def _piece(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece__piece(self)
++
++- def surface(self):
++- return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_surface(self)
+++ def _surface(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece__surface(self)
++
++- def id(self):
++- return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_id(self)
+++ def _id(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece__id(self)
++
++- def begin(self):
++- return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_begin(self)
+++ def _begin(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece__begin(self)
+++
+++ def _end(self):
+++ return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece__end(self)
+++
+++ piece = property(_piece)
+++ surface = property(_surface)
+++ id = property(_id)
+++ begin = property(_begin)
+++ end = property(_end)
+++
+++ def __str__(self):
+++ return ('piece: \"{}\"\n'
+++ 'id: {}\n'
+++ 'surface: \"{}\"\n'
+++ 'begin: {}\n'
+++ 'end: {}\n').format(self.piece, self.id, self.surface,
+++ self.begin, self.end)
+++ __repr__ = __str__
++
++- def end(self):
++- return _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_end(self)
++
++ # Register ImmutableSentencePieceText_ImmutableSentencePiece in _sentencepiece:
++ _sentencepiece.ImmutableSentencePieceText_ImmutableSentencePiece_swigregister(ImmutableSentencePieceText_ImmutableSentencePiece)
++@@ -95,32 +111,45 @@ class ImmutableSentencePieceText(object):
++ _sentencepiece.ImmutableSentencePieceText_swiginit(self, _sentencepiece.new_ImmutableSentencePieceText())
++ __swig_destroy__ = _sentencepiece.delete_ImmutableSentencePieceText
++
++- def pieces_size(self):
++- return _sentencepiece.ImmutableSentencePieceText_pieces_size(self)
+++ def _pieces_size(self):
+++ return _sentencepiece.ImmutableSentencePieceText__pieces_size(self)
++
++- def pieces(self, index):
++- return _sentencepiece.ImmutableSentencePieceText_pieces(self, index)
+++ def _pieces(self, index):
+++ return _sentencepiece.ImmutableSentencePieceText__pieces(self, index)
++
++- def text(self):
++- return _sentencepiece.ImmutableSentencePieceText_text(self)
+++ def _text(self):
+++ return _sentencepiece.ImmutableSentencePieceText__text(self)
++
++- def score(self):
++- return _sentencepiece.ImmutableSentencePieceText_score(self)
+++ def _score(self):
+++ return _sentencepiece.ImmutableSentencePieceText__score(self)
++
++ def SerializeAsString(self):
++ return _sentencepiece.ImmutableSentencePieceText_SerializeAsString(self)
++
++- def _pieces(self, index):
++- return _sentencepiece.ImmutableSentencePieceText__pieces(self, index)
+++ text = property(_text)
+++ score = property(_score)
++
++- def pieces(self, i):
++- return self._pieces(i)
+++ class ImmutableSentencePieceIterator:
+++ def __init__(self, proto):
+++ self.proto = proto
+++ self.len = self.proto._pieces_size()
++
++- def __len__(self):
++- return self.pieces_size()
+++ def __len__(self):
+++ return self.len
+++
+++ def __getitem__(self, index):
+++ if index < 0 or index >= self.len:
+++ raise IndexError('piece index is out of range')
+++ return self.proto._pieces(index)
+++
+++ def __str__(self):
+++ return '\n'.join(['pieces {{\n{}}}'.format(str(x)) for x in self])
+++
+++ __repr__ = __str__
++
++- def __getitem__(self, i):
++- return self._pieces(i)
+++ @property
+++ def pieces(self):
+++ return ImmutableSentencePieceText.ImmutableSentencePieceIterator(self)
++
++ def __eq__(self, other):
++ return self.SerializeAsString() == other.SerializeAsString()
++@@ -128,6 +157,14 @@ class ImmutableSentencePieceText(object):
++ def __hash__(self):
++ return hash(self.SerializeAsString())
++
+++ def __str__(self):
+++ return ('text: \"{}\"\n'
+++ 'score: {}\n'
+++ '{}').format(self.text, self.score,
+++ '\n'.join(['pieces {{\n{}}}'.format(str(x)) for x in self.pieces]))
+++
+++ __repr__ = __str__
+++
++
++ # Register ImmutableSentencePieceText in _sentencepiece:
++ _sentencepiece.ImmutableSentencePieceText_swigregister(ImmutableSentencePieceText)
++@@ -140,26 +177,36 @@ class ImmutableNBestSentencePieceText(object):
++ _sentencepiece.ImmutableNBestSentencePieceText_swiginit(self, _sentencepiece.new_ImmutableNBestSentencePieceText())
++ __swig_destroy__ = _sentencepiece.delete_ImmutableNBestSentencePieceText
++
++- def nbests_size(self):
++- return _sentencepiece.ImmutableNBestSentencePieceText_nbests_size(self)
+++ def _nbests_size(self):
+++ return _sentencepiece.ImmutableNBestSentencePieceText__nbests_size(self)
++
++- def nbests(self, index):
++- return _sentencepiece.ImmutableNBestSentencePieceText_nbests(self, index)
+++ def _nbests(self, index):
+++ return _sentencepiece.ImmutableNBestSentencePieceText__nbests(self, index)
++
++ def SerializeAsString(self):
++ return _sentencepiece.ImmutableNBestSentencePieceText_SerializeAsString(self)
++
++- def _nbests(self, index):
++- return _sentencepiece.ImmutableNBestSentencePieceText__nbests(self, index)
+++ class ImmutableSentencePieceTextIterator:
+++ def __init__(self, proto):
+++ self.proto = proto
+++ self.len = self.proto._nbests_size()
++
++- def __nbests__(self, i):
++- return self._nbests(i)
+++ def __len__(self):
+++ return self.len
++
++- def __len__(self):
++- return self.nbests_size()
+++ def __getitem__(self, index):
+++ if index < 0 or index >= self.len:
+++ raise IndexError('nbests index is out of range')
+++ return self.proto._nbests(index)
+++
+++ def __str__(self):
+++ return '\n'.join(['nbests {{\n{}}}'.format(str(x)) for x in self])
+++
+++ __repr__ = __str__
++
++- def __getitem__(self, i):
++- return self._nbests(i)
+++ @property
+++ def nbests(self):
+++ return ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator(self)
++
++ def __eq__(self, other):
++ return self.SerializeAsString() == other.SerializeAsString()
++@@ -167,6 +214,11 @@ class ImmutableNBestSentencePieceText(object):
++ def __hash__(self):
++ return hash(self.SerializeAsString())
++
+++ def __str__(self):
+++ return '\n'.join(['nbests {{\n{}}}'.format(str(x)) for x in self.nbests])
+++
+++ __repr__ = __str__
+++
++
++ # Register ImmutableNBestSentencePieceText in _sentencepiece:
++ _sentencepiece.ImmutableNBestSentencePieceText_swigregister(ImmutableNBestSentencePieceText)
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 1a94fef..8309fc2 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -1239,60 +1239,117 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ }
++ }
++
+++%extend sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece {
+++ %rename(_piece) piece;
+++ %rename(_id) id;
+++ %rename(_surface) surface;
+++ %rename(_begin) begin;
+++ %rename(_end) end;
+++
+++ %pythoncode %{
+++ piece = property(_piece)
+++ surface = property(_surface)
+++ id = property(_id)
+++ begin = property(_begin)
+++ end = property(_end)
+++
+++ def __str__(self):
+++ return ('piece: \"{}\"\n'
+++ 'id: {}\n'
+++ 'surface: \"{}\"\n'
+++ 'begin: {}\n'
+++ 'end: {}\n').format(self.piece, self.id, self.surface,
+++ self.begin, self.end)
+++ __repr__ = __str__
+++ %}
+++}
+++
++ %extend sentencepiece::ImmutableSentencePieceText {
++- ImmutableSentencePieceText_ImmutableSentencePiece _pieces(int index) const {
++- if (index < 0 || index >= static_cast<int>($self->pieces_size())) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "piece index is out of range.");
++- }
++- return $self->pieces(index);
++- }
+++ %rename(_text) text;
+++ %rename(_score) score;
+++ %rename(_pieces) pieces;
+++ %rename(_pieces_size) pieces_size;
+++
+++ %pythoncode %{
+++ text = property(_text)
+++ score = property(_score)
+++
+++ class ImmutableSentencePieceIterator:
+++ def __init__(self, proto):
+++ self.proto = proto
+++ self.len = self.proto._pieces_size()
+++
+++ def __len__(self):
+++ return self.len
+++
+++ def __getitem__(self, index):
+++ if index < 0 or index >= self.len:
+++ raise IndexError('piece index is out of range')
+++ return self.proto._pieces(index)
+++
+++ def __str__(self):
+++ return '\n'.join(['pieces {{\n{}}}'.format(str(x)) for x in self])
+++
+++ __repr__ = __str__
+++
+++ @property
+++ def pieces(self):
+++ return ImmutableSentencePieceText.ImmutableSentencePieceIterator(self)
+++
+++ def __eq__(self, other):
+++ return self.SerializeAsString() == other.SerializeAsString()
+++
+++ def __hash__(self):
+++ return hash(self.SerializeAsString())
+++
+++ def __str__(self):
+++ return ('text: \"{}\"\n'
+++ 'score: {}\n'
+++ '{}').format(self.text, self.score,
+++ '\n'.join(['pieces {{\n{}}}'.format(str(x)) for x in self.pieces]))
+++
+++ __repr__ = __str__
+++ %}
+++}
++
++-%pythoncode {
++- def pieces(self, i):
++- return self._pieces(i)
+++%extend sentencepiece::ImmutableNBestSentencePieceText {
+++ %rename(_nbests) nbests;
+++ %rename(_nbests_size) nbests_size;
++
++- def __len__(self):
++- return self.pieces_size()
+++ %pythoncode %{
+++ class ImmutableSentencePieceTextIterator:
+++ def __init__(self, proto):
+++ self.proto = proto
+++ self.len = self.proto._nbests_size()
++
++- def __getitem__(self, i):
++- return self._pieces(i)
+++ def __len__(self):
+++ return self.len
++
++- def __eq__(self, other):
++- return self.SerializeAsString() == other.SerializeAsString()
+++ def __getitem__(self, index):
+++ if index < 0 or index >= self.len:
+++ raise IndexError('nbests index is out of range')
+++ return self.proto._nbests(index)
++
++- def __hash__(self):
++- return hash(self.SerializeAsString())
++-}
++-}
++-
++-%extend sentencepiece::ImmutableNBestSentencePieceText {
++- ImmutableSentencePieceText _nbests(int index) const {
++- if (index < 0 || index >= static_cast<int>($self->nbests_size())) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "nbest index is out of range.");
++- }
++- return $self->nbests(index);
++- }
+++ def __str__(self):
+++ return '\n'.join(['nbests {{\n{}}}'.format(str(x)) for x in self])
++
++-%pythoncode {
++- def __nbests__(self, i):
++- return self._nbests(i)
+++ __repr__ = __str__
++
++- def __len__(self):
++- return self.nbests_size()
+++ @property
+++ def nbests(self):
+++ return ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator(self)
+++
+++ def __eq__(self, other):
+++ return self.SerializeAsString() == other.SerializeAsString()
++
++- def __getitem__(self, i):
++- return self._nbests(i)
+++ def __hash__(self):
+++ return hash(self.SerializeAsString())
++
++- def __eq__(self, other):
++- return self.SerializeAsString() == other.SerializeAsString()
+++ def __str__(self):
+++ return '\n'.join(['nbests {{\n{}}}'.format(str(x)) for x in self.nbests])
++
++- def __hash__(self):
++- return hash(self.SerializeAsString())
++-}
+++ __repr__ = __str__
+++ %}
++ }
++
++ %typemap(out) std::vector<int> {
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 4b8b5ef..0a8df5f 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -3299,22 +3299,6 @@ SWIG_From_float (float value)
++ return SWIG_From_double (value);
++ }
++
++-SWIGINTERN sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece sentencepiece_ImmutableSentencePieceText__pieces(sentencepiece::ImmutableSentencePieceText const *self,int index){
++- if (index < 0 || index >= static_cast<int>(self->pieces_size())) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "piece index is out of range.");
++- }
++- return self->pieces(index);
++- }
++-SWIGINTERN sentencepiece::ImmutableSentencePieceText sentencepiece_ImmutableNBestSentencePieceText__nbests(sentencepiece::ImmutableNBestSentencePieceText const *self,int index){
++- if (index < 0 || index >= static_cast<int>(self->nbests_size())) {
++- throw sentencepiece::util::Status(
++- sentencepiece::util::StatusCode::kOutOfRange,
++- "nbest index is out of range.");
++- }
++- return self->nbests(index);
++- }
++
++ SWIGINTERN swig_type_info*
++ SWIG_pchar_descriptor(void)
++@@ -3846,7 +3830,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_piece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece__piece(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
++ void *argp1 = 0 ;
++@@ -3858,7 +3842,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_pie
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_piece" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece__piece" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
++ {
++@@ -3880,7 +3864,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_surface(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece__surface(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
++ void *argp1 = 0 ;
++@@ -3892,7 +3876,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_sur
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_surface" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece__surface" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
++ {
++@@ -3914,7 +3898,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece__id(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
++ void *argp1 = 0 ;
++@@ -3926,7 +3910,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_id(
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_id" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece__id" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
++ {
++@@ -3945,7 +3929,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_begin(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece__begin(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
++ void *argp1 = 0 ;
++@@ -3957,7 +3941,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_beg
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_begin" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece__begin" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
++ {
++@@ -3976,7 +3960,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_end(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece__end(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *arg1 = (sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece *) 0 ;
++ void *argp1 = 0 ;
++@@ -3988,7 +3972,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_ImmutableSentencePiece_end
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece_end" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_ImmutableSentencePiece__end" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece * >(argp1);
++ {
++@@ -4069,7 +4053,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces_size(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText__pieces_size(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++ void *argp1 = 0 ;
++@@ -4081,7 +4065,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces_size(PyObject *SWIG
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_pieces_size" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText__pieces_size" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
++ {
++@@ -4100,7 +4084,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText__pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++ int arg2 ;
++@@ -4111,15 +4095,15 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_pieces(PyObject *SWIGUNUSE
++ PyObject *swig_obj[2] ;
++ sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece result;
++
++- if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText_pieces", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText__pieces", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText__pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++ if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText_pieces" "', argument " "2"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText__pieces" "', argument " "2"" of type '" "int""'");
++ }
++ arg2 = static_cast< int >(val2);
++ {
++@@ -4138,7 +4122,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_text(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText__text(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++ void *argp1 = 0 ;
++@@ -4150,7 +4134,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_text(PyObject *SWIGUNUSEDP
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_text" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText__text" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
++ {
++@@ -4172,7 +4156,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_score(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText__score(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++ void *argp1 = 0 ;
++@@ -4184,7 +4168,7 @@ SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText_score(PyObject *SWIGUNUSED
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText_score" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText__score" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
++ {
++@@ -4236,44 +4220,6 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableSentencePieceText__pieces(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++- PyObject *resultobj = 0;
++- sentencepiece::ImmutableSentencePieceText *arg1 = (sentencepiece::ImmutableSentencePieceText *) 0 ;
++- int arg2 ;
++- void *argp1 = 0 ;
++- int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece result;
++-
++- if (!SWIG_Python_UnpackTuple(args, "ImmutableSentencePieceText__pieces", 2, 2, swig_obj)) SWIG_fail;
++- res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, 0 | 0 );
++- if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableSentencePieceText__pieces" "', argument " "1"" of type '" "sentencepiece::ImmutableSentencePieceText const *""'");
++- }
++- arg1 = reinterpret_cast< sentencepiece::ImmutableSentencePieceText * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableSentencePieceText__pieces" "', argument " "2"" of type '" "int""'");
++- }
++- arg2 = static_cast< int >(val2);
++- {
++- try {
++- result = sentencepiece_ImmutableSentencePieceText__pieces((sentencepiece::ImmutableSentencePieceText const *)arg1,arg2);
++- ReleaseResultObject(resultobj);
++- }
++- catch (const sentencepiece::util::Status &status) {
++- SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++- }
++- }
++- resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece(static_cast< const sentencepiece::ImmutableSentencePieceText_ImmutableSentencePiece& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText_ImmutableSentencePiece, SWIG_POINTER_OWN | 0 );
++- return resultobj;
++-fail:
++- return NULL;
++-}
++-
++-
++ SWIGINTERN PyObject *ImmutableSentencePieceText_swigregister(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *obj;
++ if (!SWIG_Python_UnpackTuple(args, "swigregister", 1, 1, &obj)) return NULL;
++@@ -4336,7 +4282,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests_size(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText__nbests_size(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
++ void *argp1 = 0 ;
++@@ -4348,7 +4294,7 @@ SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests_size(PyObject
++ swig_obj[0] = args;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_nbests_size" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText__nbests_size" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
++ {
++@@ -4367,7 +4313,7 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
+++SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText__nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *resultobj = 0;
++ sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
++ int arg2 ;
++@@ -4378,15 +4324,15 @@ SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText_nbests(PyObject *SWIG
++ PyObject *swig_obj[2] ;
++ sentencepiece::ImmutableSentencePieceText result;
++
++- if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText_nbests", 2, 2, swig_obj)) SWIG_fail;
+++ if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText__nbests", 2, 2, swig_obj)) SWIG_fail;
++ res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
++ if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
+++ SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText__nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
++ }
++ arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
++ ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++ if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText_nbests" "', argument " "2"" of type '" "int""'");
+++ SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText__nbests" "', argument " "2"" of type '" "int""'");
++ }
++ arg2 = static_cast< int >(val2);
++ {
++@@ -4438,44 +4384,6 @@ fail:
++ }
++
++
++-SWIGINTERN PyObject *_wrap_ImmutableNBestSentencePieceText__nbests(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++- PyObject *resultobj = 0;
++- sentencepiece::ImmutableNBestSentencePieceText *arg1 = (sentencepiece::ImmutableNBestSentencePieceText *) 0 ;
++- int arg2 ;
++- void *argp1 = 0 ;
++- int res1 = 0 ;
++- int val2 ;
++- int ecode2 = 0 ;
++- PyObject *swig_obj[2] ;
++- sentencepiece::ImmutableSentencePieceText result;
++-
++- if (!SWIG_Python_UnpackTuple(args, "ImmutableNBestSentencePieceText__nbests", 2, 2, swig_obj)) SWIG_fail;
++- res1 = SWIG_ConvertPtr(swig_obj[0], &argp1,SWIGTYPE_p_sentencepiece__ImmutableNBestSentencePieceText, 0 | 0 );
++- if (!SWIG_IsOK(res1)) {
++- SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "ImmutableNBestSentencePieceText__nbests" "', argument " "1"" of type '" "sentencepiece::ImmutableNBestSentencePieceText const *""'");
++- }
++- arg1 = reinterpret_cast< sentencepiece::ImmutableNBestSentencePieceText * >(argp1);
++- ecode2 = SWIG_AsVal_int(swig_obj[1], &val2);
++- if (!SWIG_IsOK(ecode2)) {
++- SWIG_exception_fail(SWIG_ArgError(ecode2), "in method '" "ImmutableNBestSentencePieceText__nbests" "', argument " "2"" of type '" "int""'");
++- }
++- arg2 = static_cast< int >(val2);
++- {
++- try {
++- result = sentencepiece_ImmutableNBestSentencePieceText__nbests((sentencepiece::ImmutableNBestSentencePieceText const *)arg1,arg2);
++- ReleaseResultObject(resultobj);
++- }
++- catch (const sentencepiece::util::Status &status) {
++- SWIG_exception(ToSwigError(status.code()), status.ToString().c_str());
++- }
++- }
++- resultobj = SWIG_NewPointerObj((new sentencepiece::ImmutableSentencePieceText(static_cast< const sentencepiece::ImmutableSentencePieceText& >(result))), SWIGTYPE_p_sentencepiece__ImmutableSentencePieceText, SWIG_POINTER_OWN | 0 );
++- return resultobj;
++-fail:
++- return NULL;
++-}
++-
++-
++ SWIGINTERN PyObject *ImmutableNBestSentencePieceText_swigregister(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
++ PyObject *obj;
++ if (!SWIG_Python_UnpackTuple(args, "swigregister", 1, 1, &obj)) return NULL;
++@@ -8475,29 +8383,27 @@ static PyMethodDef SwigMethods[] = {
++ { "SWIG_PyInstanceMethod_New", SWIG_PyInstanceMethod_New, METH_O, NULL},
++ { "new_ImmutableSentencePieceText_ImmutableSentencePiece", _wrap_new_ImmutableSentencePieceText_ImmutableSentencePiece, METH_NOARGS, NULL},
++ { "delete_ImmutableSentencePieceText_ImmutableSentencePiece", _wrap_delete_ImmutableSentencePieceText_ImmutableSentencePiece, METH_O, NULL},
++- { "ImmutableSentencePieceText_ImmutableSentencePiece_piece", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_piece, METH_O, NULL},
++- { "ImmutableSentencePieceText_ImmutableSentencePiece_surface", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_surface, METH_O, NULL},
++- { "ImmutableSentencePieceText_ImmutableSentencePiece_id", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_id, METH_O, NULL},
++- { "ImmutableSentencePieceText_ImmutableSentencePiece_begin", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_begin, METH_O, NULL},
++- { "ImmutableSentencePieceText_ImmutableSentencePiece_end", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece_end, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece__piece", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece__piece, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece__surface", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece__surface, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece__id", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece__id, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece__begin", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece__begin, METH_O, NULL},
+++ { "ImmutableSentencePieceText_ImmutableSentencePiece__end", _wrap_ImmutableSentencePieceText_ImmutableSentencePiece__end, METH_O, NULL},
++ { "ImmutableSentencePieceText_ImmutableSentencePiece_swigregister", ImmutableSentencePieceText_ImmutableSentencePiece_swigregister, METH_O, NULL},
++ { "ImmutableSentencePieceText_ImmutableSentencePiece_swiginit", ImmutableSentencePieceText_ImmutableSentencePiece_swiginit, METH_VARARGS, NULL},
++ { "new_ImmutableSentencePieceText", _wrap_new_ImmutableSentencePieceText, METH_NOARGS, NULL},
++ { "delete_ImmutableSentencePieceText", _wrap_delete_ImmutableSentencePieceText, METH_O, NULL},
++- { "ImmutableSentencePieceText_pieces_size", _wrap_ImmutableSentencePieceText_pieces_size, METH_O, NULL},
++- { "ImmutableSentencePieceText_pieces", _wrap_ImmutableSentencePieceText_pieces, METH_VARARGS, NULL},
++- { "ImmutableSentencePieceText_text", _wrap_ImmutableSentencePieceText_text, METH_O, NULL},
++- { "ImmutableSentencePieceText_score", _wrap_ImmutableSentencePieceText_score, METH_O, NULL},
++- { "ImmutableSentencePieceText_SerializeAsString", _wrap_ImmutableSentencePieceText_SerializeAsString, METH_O, NULL},
+++ { "ImmutableSentencePieceText__pieces_size", _wrap_ImmutableSentencePieceText__pieces_size, METH_O, NULL},
++ { "ImmutableSentencePieceText__pieces", _wrap_ImmutableSentencePieceText__pieces, METH_VARARGS, NULL},
+++ { "ImmutableSentencePieceText__text", _wrap_ImmutableSentencePieceText__text, METH_O, NULL},
+++ { "ImmutableSentencePieceText__score", _wrap_ImmutableSentencePieceText__score, METH_O, NULL},
+++ { "ImmutableSentencePieceText_SerializeAsString", _wrap_ImmutableSentencePieceText_SerializeAsString, METH_O, NULL},
++ { "ImmutableSentencePieceText_swigregister", ImmutableSentencePieceText_swigregister, METH_O, NULL},
++ { "ImmutableSentencePieceText_swiginit", ImmutableSentencePieceText_swiginit, METH_VARARGS, NULL},
++ { "new_ImmutableNBestSentencePieceText", _wrap_new_ImmutableNBestSentencePieceText, METH_NOARGS, NULL},
++ { "delete_ImmutableNBestSentencePieceText", _wrap_delete_ImmutableNBestSentencePieceText, METH_O, NULL},
++- { "ImmutableNBestSentencePieceText_nbests_size", _wrap_ImmutableNBestSentencePieceText_nbests_size, METH_O, NULL},
++- { "ImmutableNBestSentencePieceText_nbests", _wrap_ImmutableNBestSentencePieceText_nbests, METH_VARARGS, NULL},
++- { "ImmutableNBestSentencePieceText_SerializeAsString", _wrap_ImmutableNBestSentencePieceText_SerializeAsString, METH_O, NULL},
+++ { "ImmutableNBestSentencePieceText__nbests_size", _wrap_ImmutableNBestSentencePieceText__nbests_size, METH_O, NULL},
++ { "ImmutableNBestSentencePieceText__nbests", _wrap_ImmutableNBestSentencePieceText__nbests, METH_VARARGS, NULL},
+++ { "ImmutableNBestSentencePieceText_SerializeAsString", _wrap_ImmutableNBestSentencePieceText_SerializeAsString, METH_O, NULL},
++ { "ImmutableNBestSentencePieceText_swigregister", ImmutableNBestSentencePieceText_swigregister, METH_O, NULL},
++ { "ImmutableNBestSentencePieceText_swiginit", ImmutableNBestSentencePieceText_swiginit, METH_VARARGS, NULL},
++ { "new_SentencePieceProcessor", _wrap_new_SentencePieceProcessor, METH_NOARGS, NULL},
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index 5e4af7f..ed792bd 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -305,6 +305,12 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ s4 = self.sp_.DecodePiecesAsImmutableProto(['foo', 'bar'])
++ s5 = self.sp_.DecodeIdsAsImmutableProto([20, 30])
++
+++ print(s1)
+++ print(s2)
+++ print(s3)
+++ print(s4)
+++ print(s5)
+++
++ t1 = self.sp_.encode_as_immutable_proto(text)
++ t2 = self.sp_.sample_encode_as_immutable_proto(text, 10, 0.2)
++ t3 = self.sp_.nbest_encode_as_immutable_proto(text, 10)
++@@ -339,35 +345,35 @@ class TestSentencepieceProcessor(unittest.TestCase):
++
++ v1 = self.sp_.EncodeAsIds(text)
++ v2 = self.sp_.EncodeAsPieces(text)
++- self.assertEqual([x.id() for x in s1], v1)
++- self.assertEqual([x.piece() for x in s1], v2)
++- self.assertEqual(text, s1.text())
+++ self.assertEqual([x.id for x in s1.pieces], v1)
+++ self.assertEqual([x.piece for x in s1.pieces], v2)
+++ self.assertEqual(text, s1.text)
++
++- surfaces1 = [s1.text()[x.begin():x.end()] for x in s1]
++- surfaces2 = [x.surface() for x in s1]
+++ surfaces1 = [s1.text[x.begin:x.end] for x in s1.pieces]
+++ surfaces2 = [x.surface for x in s1.pieces]
++ self.assertEqual(surfaces1, surfaces2)
++
++ ids = []
++- for i in range(s1.pieces_size()):
++- ids.append(s1.pieces(i).id())
+++ for i in range(len(s1.pieces)):
+++ ids.append(s1.pieces[i].id)
++ self.assertEqual(ids, v1)
++
++ pieces = []
++- for i in range(s1.pieces_size()):
++- pieces.append(s1.pieces(i).piece())
+++ for i in range(len(s1.pieces)):
+++ pieces.append(s1.pieces[i].piece)
++ self.assertEqual(pieces, v2)
++
++ # Japanese offset
++ s1 = self.jasp_.EncodeAsImmutableProto('吾輩は猫である。Hello world. ABC 123')
++- surfaces1 = [s1.text()[x.begin():x.end()] for x in s1]
++- surfaces2 = [x.surface() for x in s1]
+++ surfaces1 = [s1.text[x.begin:x.end] for x in s1.pieces]
+++ surfaces2 = [x.surface for x in s1.pieces]
++ self.assertEqual(surfaces1, surfaces2)
++
++- ids = [x.id() for x in s1]
+++ ids = [x.id for x in s1.pieces]
++ s2 = self.jasp_.DecodeIdsAsImmutableProto(ids)
++ self.assertEqual(s2, s1)
++
++- pieces = [x.piece() for x in s1]
+++ pieces = [x.piece for x in s1.pieces]
++ s2 = self.jasp_.DecodePiecesAsImmutableProto(pieces)
++ self.assertEqual(s2, s1)
++
++@@ -395,29 +401,29 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(sp.encode([text], out_type='serialized_proto'), [sprotos])
++ self.assertEqual(sp.encode([text], out_type='immutable_proto'), [iprotos])
++
++- self.assertEqual(len(iprotos), len(pieces))
++- self.assertEqual(len(iprotos), len(ids))
++- self.assertEqual(iprotos.text(), text)
+++ self.assertEqual(len(iprotos.pieces), len(pieces))
+++ self.assertEqual(len(iprotos.pieces), len(ids))
+++ self.assertEqual(iprotos.text, text)
++
++- self.assertEqual(len(iprotos2), len(pieces2))
++- self.assertEqual(len(iprotos2), len(ids2))
++- self.assertEqual(iprotos2.text(), text2)
+++ self.assertEqual(len(iprotos2.pieces), len(pieces2))
+++ self.assertEqual(len(iprotos2.pieces), len(ids2))
+++ self.assertEqual(iprotos2.text, text2)
++
++- for i in range(len(iprotos)):
++- self.assertEqual(ids[i], iprotos.pieces(i).id())
++- self.assertEqual(pieces[i], iprotos.pieces(i).piece())
+++ for i in range(len(iprotos.pieces)):
+++ self.assertEqual(ids[i], iprotos.pieces[i].id)
+++ self.assertEqual(pieces[i], iprotos.pieces[i].piece)
++
++- for i, piece in enumerate(iprotos):
++- self.assertEqual(ids[i], piece.id())
++- self.assertEqual(pieces[i], piece.piece())
+++ for i, piece in enumerate(iprotos.pieces):
+++ self.assertEqual(ids[i], piece.id)
+++ self.assertEqual(pieces[i], piece.piece)
++
++- for i in range(len(iprotos2)):
++- self.assertEqual(ids2[i], iprotos2.pieces(i).id())
++- self.assertEqual(pieces2[i], iprotos2.pieces(i).piece())
+++ for i in range(len(iprotos2.pieces)):
+++ self.assertEqual(ids2[i], iprotos2.pieces[i].id)
+++ self.assertEqual(pieces2[i], iprotos2.pieces[i].piece)
++
++- for i, piece in enumerate(iprotos2):
++- self.assertEqual(ids2[i], piece.id())
++- self.assertEqual(pieces2[i], piece.piece())
+++ for i, piece in enumerate(iprotos2.pieces):
+++ self.assertEqual(ids2[i], piece.id)
+++ self.assertEqual(pieces2[i], piece.piece)
++
++ detok_ids = self.sp_.DecodeIds(ids)
++ detok_pieces = self.sp_.DecodePieces(pieces)
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Fri, 5 Aug 2022 14:47:02 +0900
++Subject: automatically detect the number of CPUs in batch processing.
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/__init__.py | 27 +++++++++++++--------
++ python/src/sentencepiece/sentencepiece.i | 32 ++++++++++++++++---------
++ python/src/sentencepiece/sentencepiece_wrap.cxx | 5 +++-
++ python/test/sentencepiece_test.py | 32 +++++++++++++++++++++++++
++ 4 files changed, 74 insertions(+), 22 deletions(-)
++
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index 12dc631..ce9d60d 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -97,6 +97,13 @@ class ImmutableSentencePieceText_ImmutableSentencePiece(object):
++ 'begin: {}\n'
++ 'end: {}\n').format(self.piece, self.id, self.surface,
++ self.begin, self.end)
+++
+++ def __eq__(self, other):
+++ return self.piece == other.piece and self.id == other.id and self.surface == other.surface and self.begin == other.begin and self.end == other.end
+++
+++ def __hash__(self):
+++ return hash(str(self))
+++
++ __repr__ = __str__
++
++
++@@ -395,7 +402,7 @@ class SentencePieceProcessor(object):
++ enable_sampling=False,
++ nbest_size=-1,
++ alpha=0.1,
++- num_threads=1):
+++ num_threads=-1):
++ """Initialzie sentencepieceProcessor.
++
++ Args:
++@@ -407,15 +414,15 @@ class SentencePieceProcessor(object):
++ reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++ emit_unk_piece: Emits the unk literal string (Default = false)
++- nbest_size: sampling parameters for unigram. Invalid for BPE-Dropout.
+++ nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout.
++ nbest_size = {0,1}: No sampling is performed.
++ nbest_size > 1: samples from the nbest_size results.
++ nbest_size < 0: assuming that nbest_size is infinite and samples
++ from the all hypothesis (lattice) using
++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and dropout probability of
++- merge operations for BPE-dropout.
++- num_threads: number of threads in batch processing.
+++ merge operations for BPE-dropout.
+++ num_threads: number of threads in batch processing (Default = -1, auto-detected)
++ """
++
++ _sentencepiece_processor_init_native(self)
++@@ -450,18 +457,18 @@ class SentencePieceProcessor(object):
++ out_type: output type. int or str.
++ add_bos: Add <s> to the result (Default = false)
++ add_eos: Add </s> to the result (Default = false) <s>/</s> is added after
++- reversing (if enabled).
+++ reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++ emit_unk_piece: Emits the unk literal string (Default = false)
++- nbest_size: sampling parameters for unigram. Invalid for BPE-Dropout.
+++ nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout.
++ nbest_size = {0,1}: No sampling is performed.
++ nbest_size > 1: samples from the nbest_size results.
++ nbest_size < 0: assuming that nbest_size is infinite and samples
++- from the all hypothesis (lattice) using
++- forward-filtering-and-backward-sampling algorithm.
+++ from the all hypothesis (lattice) using
+++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and merge probability for
++ BPE-dropout (probablity 'p' in BPE-dropout paper).
++- num_threads: the number of threads used in the batch processin (Default = 1).
+++ num_threads: the number of threads used in the batch processing (Default = -1).
++ """
++
++ if out_type is None:
++@@ -722,7 +729,7 @@ class SentencePieceProcessor(object):
++
++ Args:
++ out_type: output type. str or 'serialized_proto' or 'immutable_proto' (Default = str)
++- num_threads: the number of threads used in the batch processin (Default = 1).
+++ num_threads: the number of threads used in the batch processing (Default = -1).
++ """
++
++ if num_threads is None:
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 8309fc2..e22f763 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -233,9 +233,12 @@ class ThreadPool {
++
++ template <typename T>
++ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
+++ if (*num_threads < 0) {
+++ *num_threads = std::thread::hardware_concurrency();
+++ }
++ *num_threads = std::max<int>(1,
++ std::min<int>({*num_threads,
++- static_cast<int>(ins.size()), 256}));
+++ static_cast<int>(ins.size()), 256}));
++ }
++
++ #define DEFINE_ENCODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
++@@ -675,7 +678,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ enable_sampling=False,
++ nbest_size=-1,
++ alpha=0.1,
++- num_threads=1):
+++ num_threads=-1):
++ """Initialzie sentencepieceProcessor.
++
++ Args:
++@@ -687,15 +690,15 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++ emit_unk_piece: Emits the unk literal string (Default = false)
++- nbest_size: sampling parameters for unigram. Invalid for BPE-Dropout.
+++ nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout.
++ nbest_size = {0,1}: No sampling is performed.
++ nbest_size > 1: samples from the nbest_size results.
++ nbest_size < 0: assuming that nbest_size is infinite and samples
++ from the all hypothesis (lattice) using
++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and dropout probability of
++- merge operations for BPE-dropout.
++- num_threads: number of threads in batch processing.
+++ merge operations for BPE-dropout.
+++ num_threads: number of threads in batch processing (Default = -1, auto-detected)
++ """
++
++ _sentencepiece_processor_init_native(self)
++@@ -730,18 +733,18 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ out_type: output type. int or str.
++ add_bos: Add <s> to the result (Default = false)
++ add_eos: Add </s> to the result (Default = false) <s>/</s> is added after
++- reversing (if enabled).
+++ reversing (if enabled).
++ reverse: Reverses the tokenized sequence (Default = false)
++ emit_unk_piece: Emits the unk literal string (Default = false)
++- nbest_size: sampling parameters for unigram. Invalid for BPE-Dropout.
+++ nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout.
++ nbest_size = {0,1}: No sampling is performed.
++ nbest_size > 1: samples from the nbest_size results.
++ nbest_size < 0: assuming that nbest_size is infinite and samples
++- from the all hypothesis (lattice) using
++- forward-filtering-and-backward-sampling algorithm.
+++ from the all hypothesis (lattice) using
+++ forward-filtering-and-backward-sampling algorithm.
++ alpha: Soothing parameter for unigram sampling, and merge probability for
++ BPE-dropout (probablity 'p' in BPE-dropout paper).
++- num_threads: the number of threads used in the batch processin (Default = 1).
+++ num_threads: the number of threads used in the batch processing (Default = -1).
++ """
++
++ if out_type is None:
++@@ -1002,7 +1005,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++
++ Args:
++ out_type: output type. str or 'serialized_proto' or 'immutable_proto' (Default = str)
++- num_threads: the number of threads used in the batch processin (Default = 1).
+++ num_threads: the number of threads used in the batch processing (Default = -1).
++ """
++
++ if num_threads is None:
++@@ -1260,6 +1263,13 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ 'begin: {}\n'
++ 'end: {}\n').format(self.piece, self.id, self.surface,
++ self.begin, self.end)
+++
+++ def __eq__(self, other):
+++ return self.piece == other.piece and self.id == other.id and self.surface == other.surface and self.begin == other.begin and self.end == other.end
+++
+++ def __hash__(self):
+++ return hash(str(self))
+++
++ __repr__ = __str__
++ %}
++ }
++diff --git a/python/src/sentencepiece/sentencepiece_wrap.cxx b/python/src/sentencepiece/sentencepiece_wrap.cxx
++index 0a8df5f..1eac211 100644
++--- a/python/src/sentencepiece/sentencepiece_wrap.cxx
+++++ b/python/src/sentencepiece/sentencepiece_wrap.cxx
++@@ -3042,9 +3042,12 @@ class ThreadPool {
++
++ template <typename T>
++ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
+++ if (*num_threads < 0) {
+++ *num_threads = std::thread::hardware_concurrency();
+++ }
++ *num_threads = std::max<int>(1,
++ std::min<int>({*num_threads,
++- static_cast<int>(ins.size()), 256}));
+++ static_cast<int>(ins.size()), 256}));
++ }
++
++ #define DEFINE_ENCODE_BATCH_FUNC_IMPL(FuncName, InType, OutType) \
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index ed792bd..6cbe077 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -332,6 +332,29 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(s4, y4)
++ self.assertEqual(s5, y5)
++
+++ hset_piece = defaultdict(int)
+++
+++ # eq test
+++ for i in range(len(s1.pieces)):
+++ self.assertEqual(s1.pieces[i], t1.pieces[i])
+++ hset_piece[s1.pieces[i]] += 1
+++ hset_piece[t1.pieces[i]] += 1
+++
+++ self.assertEqual(len(hset_piece), len(s1.pieces))
+++
+++ # has test
+++ hset = defaultdict(int)
+++ hset[s1] += 1
+++ hset[t1] += 1
+++ hset[s3] += 1
+++ hset[t3] += 1
+++
+++ self.assertEqual(len(hset), 2)
+++ self.assertEqual(hset[s1], 2)
+++ self.assertEqual(hset[s3], 2)
+++ self.assertEqual(hset[t1], 2)
+++ self.assertEqual(hset[t3], 2)
+++
++ x1 = self.sp_.encode_as_serialized_proto(text)
++ x2 = self.sp_.sample_encode_as_serialized_proto(text, 10, 0.2)
++ x3 = self.sp_.nbest_encode_as_serialized_proto(text, 10)
++@@ -363,6 +386,15 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ pieces.append(s1.pieces[i].piece)
++ self.assertEqual(pieces, v2)
++
+++ for v in s3.nbests:
+++ self.assertEqual(text, v.text)
+++ self.assertEqual(self.sp_.Decode([x.id for x in v.pieces]), text)
+++
+++ for i in range(len(s3.nbests)):
+++ self.assertEqual(text, s3.nbests[i].text)
+++ self.assertEqual(
+++ self.sp_.Decode([x.id for x in s3.nbests[i].pieces]), text)
+++
++ # Japanese offset
++ s1 = self.jasp_.EncodeAsImmutableProto('吾輩は猫である。Hello world. ABC 123')
++ surfaces1 = [s1.text[x.begin:x.end] for x in s1.pieces]
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Fri, 5 Aug 2022 16:34:44 +0900
++Subject: support slice in pieces/nbests objects
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/src/sentencepiece/__init__.py | 8 ++++++++
++ python/src/sentencepiece/sentencepiece.i | 8 ++++++++
++ python/test/sentencepiece_test.py | 4 ++++
++ 3 files changed, 20 insertions(+)
++
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index ce9d60d..cf06830 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -145,6 +145,10 @@ class ImmutableSentencePieceText(object):
++ return self.len
++
++ def __getitem__(self, index):
+++ if isinstance(index, slice):
+++ return [self.proto._pieces(i) for i in range(self.len)][index.start:index.stop:index.step]
+++ if index < 0:
+++ index = index + self.len
++ if index < 0 or index >= self.len:
++ raise IndexError('piece index is out of range')
++ return self.proto._pieces(index)
++@@ -202,6 +206,10 @@ class ImmutableNBestSentencePieceText(object):
++ return self.len
++
++ def __getitem__(self, index):
+++ if isinstance(index, slice):
+++ return [self.proto._nbests(i) for i in range(self.len)][index.start:index.stop:index.step]
+++ if index < 0:
+++ index = index + self.len
++ if index < 0 or index >= self.len:
++ raise IndexError('nbests index is out of range')
++ return self.proto._nbests(index)
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index e22f763..2ac68a8 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -1293,6 +1293,10 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return self.len
++
++ def __getitem__(self, index):
+++ if isinstance(index, slice):
+++ return [self.proto._pieces(i) for i in range(self.len)][index.start:index.stop:index.step]
+++ if index < 0:
+++ index = index + self.len
++ if index < 0 or index >= self.len:
++ raise IndexError('piece index is out of range')
++ return self.proto._pieces(index)
++@@ -1336,6 +1340,10 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return self.len
++
++ def __getitem__(self, index):
+++ if isinstance(index, slice):
+++ return [self.proto._nbests(i) for i in range(self.len)][index.start:index.stop:index.step]
+++ if index < 0:
+++ index = index + self.len
++ if index < 0 or index >= self.len:
++ raise IndexError('nbests index is out of range')
++ return self.proto._nbests(index)
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index 6cbe077..92327ac 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -395,6 +395,10 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ self.assertEqual(
++ self.sp_.Decode([x.id for x in s3.nbests[i].pieces]), text)
++
+++ # slice
+++ self.assertEqual(s1.pieces[::-1], list(reversed(s1.pieces)))
+++ self.assertEqual(s3.nbests[::-1], list(reversed(s3.nbests)))
+++
++ # Japanese offset
++ s1 = self.jasp_.EncodeAsImmutableProto('吾輩は猫である。Hello world. ABC 123')
++ surfaces1 = [s1.text[x.begin:x.end] for x in s1.pieces]
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Fri, 5 Aug 2022 19:05:52 +0900
++Subject: Updated the document
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ README.md | 1 -
++ doc/api.md | 22 ++--
++ doc/options.md | 102 ++++++++++---------
++ python/README.md | 168 +++++++++++++------------------
++ python/src/sentencepiece/__init__.py | 22 +++-
++ python/src/sentencepiece/sentencepiece.i | 22 +++-
++ python/test/sentencepiece_test.py | 20 +++-
++ 7 files changed, 199 insertions(+), 158 deletions(-)
++
++diff --git a/README.md b/README.md
++index dc71b64..1986047 100644
++--- a/README.md
+++++ b/README.md
++@@ -276,6 +276,5 @@ Then segment train/test corpus with ```--vocabulary``` option
++ * [Use custom text normalization rules](doc/normalization.md)
++ * [Use custom symbols](doc/special_symbols.md)
++ * [Python Module](python/README.md)
++-* [TensorFlow Module](tensorflow/README.md)
++ * [Segmentation and training algorithms in detail]
++
++diff --git a/doc/api.md b/doc/api.md
++index 797074c..ebde880 100644
++--- a/doc/api.md
+++++ b/doc/api.md
++@@ -14,9 +14,9 @@ if (!status.ok()) {
++ // error
++ }
++
++-// You can also load a model from std::ifstream.
++-// std::ifstream in("//path/to/model.model");
++-// auto status = processor.Load(in);
+++// You can also load a serialized model from std::string.
+++// const std::stirng str = // Load blob contents from a file.
+++// auto status = processor.LoadFromSerializedProto(str);
++ ```
++
++ ## Tokenize text (preprocessing)
++@@ -75,16 +75,20 @@ Calls `SentencePieceTrainer::Train` function to train sentencepiece model. You c
++ sentencepiece::SentencePieceTrainer::Train("--input=test/botchan.txt --model_prefix=m --vocab_size=1000");
++ ```
++
++-## SentencePieceText proto
++-You will want to use `SentencePieceText` class to obtain the pieces and ids at the same time. This proto also encodes a utf8-byte offset of each piece over user input or detokenized text.
+++## ImmutableSentencePieceText
+++You will want to use `ImmutableSentencePieceText` class to obtain the pieces and ids at the same time.
+++This proto also encodes a utf8-byte offset of each piece over user input or detokenized text.
++
++ ```C++
++-#include <sentencepiece.pb.h>
+++#include <sentencepiece_processor.h>
++
++-sentencepiece::SentencePieceText spt;
+++sentencepiece::ImmutableSentencePieceText spt;
++
++ // Encode
++-processor.Encode("This is a test.", &spt);
+++processor.Encode("This is a test.", spt.mutable_proto());
+++
+++// or
+++// spt = processor.EncodeAsImmutableProto("This is a test.");
++
++ std::cout << spt.text() << std::endl; // This is the same as the input.
++ for (const auto &piece : spt.pieces()) {
++@@ -96,7 +100,7 @@ for (const auto &piece : spt.pieces()) {
++ }
++
++ // Decode
++-processor.Decode({10, 20, 30}, &spt);
+++processor.Decode({10, 20, 30}, spt.mutable_proto());
++ std::cout << spt.text() << std::endl; // This is the same as the decoded string.
++ for (const auto &piece : spt.pieces()) {
++ // the same as above.
++diff --git a/doc/options.md b/doc/options.md
++index 26cf681..6cdc0f9 100644
++--- a/doc/options.md
+++++ b/doc/options.md
++@@ -3,50 +3,60 @@
++ The training options for the `spm_train` can be listed using `spm_train --help`. Since the standard `pip install` of sentencepiece does not necessarily install `spm_train`, the options are also listed here.
++
++ ```
++---help (show help) type: bool default: false
++---version (show version) type: bool default: false
++---minloglevel (Messages logged at a lower level than this don't actually get logged anywhere) type: int default: 0
++---input (comma separated list of input sentences) type: std::string default: ""
++---input_format (Input format. Supported format is `text` or `tsv`.) type: std::string default: ""
++---model_prefix (output model prefix) type: std::string default: ""
++---model_type (model algorithm: unigram, bpe, word or char) type: std::string default: "unigram"
++---vocab_size (vocabulary size) type: int32 default: 8000
++---accept_language (comma-separated list of languages this model can accept) type: std::string default: ""
++---self_test_sample_size (the size of self test samples) type: int32 default: 0
++---character_coverage (character coverage to determine the minimum symbols) type: double default: 0.9995
++---input_sentence_size (maximum size of sentences the trainer loads) type: int32 default: 0
++---shuffle_input_sentence (Randomly sample input sentences in advance. Valid when --input_sentence_size > 0) type: bool default: true
++---seed_sentencepiece_size (the size of seed sentencepieces) type: int32 default: 1000000
++---shrinking_factor (Keeps top shrinking_factor pieces with respect to the loss) type: double default: 0.75
++---num_threads (number of threads for training) type: int32 default: 16
++---num_sub_iterations (number of EM sub-iterations) type: int32 default: 2
++---max_sentencepiece_length (maximum length of sentence piece) type: int32 default: 16
++---max_sentence_length (maximum length of sentence in byte) type: int32 default: 4192
++---split_by_unicode_script (use Unicode script to split sentence pieces) type: bool default: true
++---split_by_number (split tokens by numbers (0-9)) type: bool default: true
++---split_by_whitespace (use a white space to split sentence pieces) type: bool default: true
++---split_digits (split all digits (0-9) into separate pieces) type: bool default: false
++---treat_whitespace_as_suffix (treat whitespace marker as suffix instead of prefix.) type: bool default: false
++---control_symbols (comma separated list of control symbols) type: std::string default: ""
++---user_defined_symbols (comma separated list of user defined symbols) type: std::string default: ""
++---required_chars (UTF8 characters in this flag are always used in the character set regardless of --character_coverage) type: std::string default: ""
++---byte_fallback (decompose unknown pieces into UTF-8 byte pieces) type: bool default: false
++---vocabulary_output_piece_score (Define score in vocab file) type: bool default: true
++---normalization_rule_name (Normalization rule name. Choose from nfkc or identity) type: std::string default: "nmt_nfkc"
++---normalization_rule_tsv (Normalization rule TSV file. ) type: std::string default: ""
++---denormalization_rule_tsv (Denormalization rule TSV file.) type: std::string default: ""
++---add_dummy_prefix (Add dummy whitespace at the beginning of text) type: bool default: true
++---remove_extra_whitespaces (Removes leading, trailing, and duplicate internal whitespace) type: bool default: true
++---hard_vocab_limit (If set to false, --vocab_size is considered as a soft limit.) type: bool default: true
++---use_all_vocab (If set to true, use all tokens as vocab. Valid for word/char models.) type: bool default: false
++---unk_id (Override UNK (<unk>) id.) type: int32 default: 0
++---bos_id (Override BOS (<s>) id. Set -1 to disable BOS.) type: int32 default: 1
++---eos_id (Override EOS (</s>) id. Set -1 to disable EOS.) type: int32 default: 2
++---pad_id (Override PAD (<pad>) id. Set -1 to disable PAD.) type: int32 default: -1
++---unk_piece (Override UNK (<unk>) piece.) type: std::string default: "<unk>"
++---bos_piece (Override BOS (<s>) piece.) type: std::string default: "<s>"
++---eos_piece (Override EOS (</s>) piece.) type: std::string default: "</s>"
++---pad_piece (Override PAD (<pad>) piece.) type: std::string default: "<pad>"
++---unk_surface (Dummy surface string for <unk>. In decoding <unk> is decoded to `unk_surface`.) type: std::string default: " ⁇ "
++---train_extremely_large_corpus (Increase bit depth for unigram tokenization.) type: bool default: false
+++Usage: ../build/src/spm_train [options] files
+++
+++ --input (comma separated list of input sentences) type: std::string default: ""
+++ --input_format (Input format. Supported format is `text` or `tsv`.) type: std::string default: ""
+++ --model_prefix (output model prefix) type: std::string default: ""
+++ --model_type (model algorithm: unigram, bpe, word or char) type: std::string default: "unigram"
+++ --vocab_size (vocabulary size) type: int32 default: 8000
+++ --accept_language (comma-separated list of languages this model can accept) type: std::string default: ""
+++ --self_test_sample_size (the size of self test samples) type: int32 default: 0
+++ --character_coverage (character coverage to determine the minimum symbols) type: double default: 0.9995
+++ --input_sentence_size (maximum size of sentences the trainer loads) type: std::uint64_t default: 0
+++ --shuffle_input_sentence (Randomly sample input sentences in advance. Valid when --input_sentence_size > 0) type: bool default: true
+++ --seed_sentencepiece_size (the size of seed sentencepieces) type: int32 default: 1000000
+++ --shrinking_factor (Keeps top shrinking_factor pieces with respect to the loss) type: double default: 0.75
+++ --num_threads (number of threads for training) type: int32 default: 16
+++ --num_sub_iterations (number of EM sub-iterations) type: int32 default: 2
+++ --max_sentencepiece_length (maximum length of sentence piece) type: int32 default: 16
+++ --max_sentence_length (maximum length of sentence in byte) type: int32 default: 4192
+++ --split_by_unicode_script (use Unicode script to split sentence pieces) type: bool default: true
+++ --split_by_number (split tokens by numbers (0-9)) type: bool default: true
+++ --split_by_whitespace (use a white space to split sentence pieces) type: bool default: true
+++ --split_digits (split all digits (0-9) into separate pieces) type: bool default: false
+++ --treat_whitespace_as_suffix (treat whitespace marker as suffix instead of prefix.) type: bool default: false
+++ --allow_whitespace_only_pieces (allow pieces that only contain (consecutive) whitespace tokens) type: bool default: false
+++ --control_symbols (comma separated list of control symbols) type: std::string default: ""
+++ --control_symbols_file (load control_symbols from file.) type: std::string default: ""
+++ --user_defined_symbols (comma separated list of user defined symbols) type: std::string default: ""
+++ --user_defined_symbols_file (load user_defined_symbols from file.) type: std::string default: ""
+++ --required_chars (UTF8 characters in this flag are always used in the character set regardless of --character_coverage) type: std::string default: ""
+++ --required_chars_file (load required_chars from file.) type: std::string default: ""
+++ --byte_fallback (decompose unknown pieces into UTF-8 byte pieces) type: bool default: false
+++ --vocabulary_output_piece_score (Define score in vocab file) type: bool default: true
+++ --normalization_rule_name (Normalization rule name. Choose from nfkc or identity) type: std::string default: "nmt_nfkc"
+++ --normalization_rule_tsv (Normalization rule TSV file. ) type: std::string default: ""
+++ --denormalization_rule_tsv (Denormalization rule TSV file.) type: std::string default: ""
+++ --add_dummy_prefix (Add dummy whitespace at the beginning of text) type: bool default: true
+++ --remove_extra_whitespaces (Removes leading, trailing, and duplicate internal whitespace) type: bool default: true
+++ --hard_vocab_limit (If set to false, --vocab_size is considered as a soft limit.) type: bool default: true
+++ --use_all_vocab (If set to true, use all tokens as vocab. Valid for word/char models.) type: bool default: false
+++ --unk_id (Override UNK (<unk>) id.) type: int32 default: 0
+++ --bos_id (Override BOS (<s>) id. Set -1 to disable BOS.) type: int32 default: 1
+++ --eos_id (Override EOS (</s>) id. Set -1 to disable EOS.) type: int32 default: 2
+++ --pad_id (Override PAD (<pad>) id. Set -1 to disable PAD.) type: int32 default: -1
+++ --unk_piece (Override UNK (<unk>) piece.) type: std::string default: "<unk>"
+++ --bos_piece (Override BOS (<s>) piece.) type: std::string default: "<s>"
+++ --eos_piece (Override EOS (</s>) piece.) type: std::string default: "</s>"
+++ --pad_piece (Override PAD (<pad>) piece.) type: std::string default: "<pad>"
+++ --unk_surface (Dummy surface string for <unk>. In decoding <unk> is decoded to `unk_surface`.) type: std::string default: " ⁇ "
+++ --train_extremely_large_corpus (Increase bit depth for unigram tokenization.) type: bool default: false
+++ --random_seed (Seed value for random generator.) type: uint32 default: 4294967295
+++ --enable_differential_privacy (Whether to add DP while training. Currently supported only by UNIGRAM model.) type: bool default: false
+++ --differential_privacy_noise_level (Amount of noise to add for DP) type: float default: 0
+++ --differential_privacy_clipping_threshold (Threshold for clipping the counts for DP) type: std::uint64_t default: 0
+++ --help (show help) type: bool default: false
+++ --version (show version) type: bool default: false
+++ --minloglevel (Messages logged at a lower level than this don't actually get logged anywhere) type: int default: 0
++ ```
++diff --git a/python/README.md b/python/README.md
++index b683082..bc5a59a 100644
++--- a/python/README.md
+++++ b/python/README.md
++@@ -9,10 +9,17 @@ For Linux (x64/i686), macOS, and Windows(win32/x64) environment, you can simply
++ % pip install sentencepiece
++ ```
++
++-To build and install the Python wrapper from source, please install [SentencePiece C++](https://github.com/google/sentencepiece#c-from-source) and try the following commands:
+++To build and install the Python wrapper from source, try the following commands to build and install wheel package.
++ ```
++-% python setup.py build
++-% sudo python setup.py install
+++% git clone https://github.com/google/sentencepiece.git
+++% cd sentencepiece
+++% mkdir build
+++% cd build
+++% cmake .. -DSPM_ENABLE_SHARED=OFF -DCMAKE_INSTALL_PREFIX=./root
+++% make install
+++% cd ../python
+++% python setup.py bdist_wheel
+++% pip install dist/sentencepiece*.whl
++ ```
++
++ If you don’t have write permission to the global site-packages directory or don’t want to install into it, please try:
++@@ -22,21 +29,50 @@ If you don’t have write permission to the global site-packages directory or do
++
++ ## Usage
++
++-See [this google colab page](https://github.com/google/sentencepiece/blob/master/python/sentencepiece_python_module_example.ipynb) to run sentencepiece interactively. (Note: this sample is written in old interface.)
+++See [this google colab page](https://github.com/google/sentencepiece/blob/master/python/sentencepiece_python_module_example.ipynb) to run sentencepiece interactively.
++
++ ### Segmentation
++ ```
++ % python
++ >>> import sentencepiece as spm
++ >>> sp = spm.SentencePieceProcessor(model_file='test/test_model.model')
+++
++ >>> sp.encode('This is a test')
++ [284, 47, 11, 4, 15, 400]
+++
++ >>> sp.encode(['This is a test', 'Hello world'], out_type=int)
++ [[284, 47, 11, 4, 15, 400], [151, 88, 21, 887]]
+++
+++>>> sp.encode_as_ids(['This is a test', 'Hello world'])
+++[[284, 47, 11, 4, 15, 400], [151, 88, 21, 887]]
+++
++ >>> sp.encode('This is a test', out_type=str)
++ ['▁This', '▁is', '▁a', '▁', 't', 'est']
+++
++ >>> sp.encode(['This is a test', 'Hello world'], out_type=str)
++ [['▁This', '▁is', '▁a', '▁', 't', 'est'], ['▁He', 'll', 'o', '▁world']]
+++
+++>>> sp.encode_as_pieces(['This is a test', 'Hello world'])
+++[['▁This', '▁is', '▁a', '▁', 't', 'est'], ['▁He', 'll', 'o', '▁world']]
+++
+++>>> proto = sp.encode('This is a test', out_type='immutable_proto')
+++>>> for n in proto.pieces:
+++... print('piece="{}" surface="{}" id={} begin={} end={}'.format(n.piece, n.surface, n.id, n.begin, n.end))
+++...
+++piece="▁This" surface="This" id=284 begin=0 end=4
+++piece="▁is" surface=" is" id=47 begin=4 end=7
+++piece="▁a" surface=" a" id=11 begin=7 end=9
+++piece="▁" surface=" " id=4 begin=9 end=10
+++piece="t" surface="t" id=15 begin=10 end=11
+++piece="est" surface="est" id=400 begin=11 end=14
+++
+++>>> [[x.id for x in proto.pieces], [x.piece for x in proto.pieces], [x.begin for x in proto.pieces], [x.end for x in proto.pieces]]
+++[[284, 47, 11, 4, 15, 400], ['▁This', '▁is', '▁a', '▁', 't', 'est'], [0, 4, 7, 9, 10, 11], [4, 7, 9, 10, 11, 14]]
+++
+++>>> proto2 = sp.encode_as_immutable_proto('This is a test')
+++>>> proto2 == proto
+++True
+++
++ >>> for _ in range(10):
++ ... sp.encode('This is a test', out_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1)
++ ...
++@@ -50,26 +86,55 @@ See [this google colab page](https://github.com/google/sentencepiece/blob/master
++ ['▁', 'T', 'h', 'is', '▁', 'is', '▁', 'a', '▁', 'te', 'st']
++ ['▁', 'This', '▁', 'i', 's', '▁a', '▁', 't', 'e', 'st']
++ ['▁This', '▁', 'is', '▁a', '▁', 't', 'est']
+++
+++>> sp.nbest_encode('This is a test', nbest_size=5, out_type=str)
+++[['▁This', '▁is', '▁a', '▁', 't', 'est'],
+++['▁This', '▁is', '▁a', '▁', 'te', 'st'],
+++['▁This', '▁is', '▁a', '▁', 'te', 's', 't'],
+++['▁This', '▁is', '▁a', '▁', 't', 'e', 'st'],
+++['▁This', '▁is', '▁a', '▁', 't', 'es', 't']]
+++
+++>>> sp.sample_encode_and_score('This is a test', num_samples=5, alpha=0.1, out_type=str, wor=True)
+++[(['▁This', '▁', 'i', 's', '▁a', '▁', 'te', 's', 't'], -3.043105125427246),
+++(['▁This', '▁', 'i', 's', '▁a', '▁', 'te', 'st'], -2.8475849628448486),
+++(['▁', 'This', '▁is', '▁', 'a', '▁', 'te', 'st'], -3.043248176574707),
+++(['▁', 'This', '▁is', '▁a', '▁', 't', 'e', 'st'], -2.87727689743042),
+++(['▁', 'This', '▁', 'i', 's', '▁', 'a', '▁', 't', 'est'], -3.6284031867980957)]
+++
++ >>> sp.decode([284, 47, 11, 4, 15, 400])
++ 'This is a test'
+++
++ >>> sp.decode([[284, 47, 11, 4, 15, 400], [151, 88, 21, 887]])
++ ['This is a test', 'Hello world']
+++
+++>>> proto = sp.decode([284, 47, 11, 4, 15, 400], out_type='immutable_proto')
+++>>> proto.text
+++'This is a test'
+++
++ >>> sp.decode(['▁', 'This', '▁', 'is', '▁a', '▁', 't', 'e', 'st'])
++ 'This is a test'
+++
++ >>> sp.decode([['▁This', '▁is', '▁a', '▁', 't', 'est'], ['▁He', 'll', 'o', '▁world']])
++ ['This is a test', 'Hello world']
+++
++ >>> sp.get_piece_size()
++ 1000
+++
++ >>> sp.id_to_piece(2)
++ '</s>'
+++
++ >>> sp.id_to_piece([2, 3, 4])
++ ['</s>', '\r', '▁']
+++
++ >>> sp.piece_to_id('<s>')
++ 1
+++
++ >>> sp.piece_to_id(['</s>', '\r', '▁'])
++ [2, 3, 4]
+++
++ >>> len(sp)
++ 1000
+++
++ >>> sp['</s>']
++ 2
++ ```
++@@ -116,98 +181,3 @@ with urllib.request.urlopen(
++ sp = spm.SentencePieceProcessor(model_proto=model.getvalue())
++ print(sp.encode('this is test'))
++ ```
++-
++-
++-### Segmentation (old interface)
++-```
++-% python
++->>> import sentencepiece as spm
++->>> sp = spm.SentencePieceProcessor()
++->>> sp.Load("test/test_model.model")
++-True
++->>> sp.EncodeAsPieces("This is a test")
++-['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'est']
++->>> sp.EncodeAsIds("This is a test")
++-[284, 47, 11, 4, 15, 400]
++->>> sp.DecodePieces(['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'est'])
++-'This is a test'
++->>> sp.NBestEncodeAsPieces("This is a test", 5)
++-[['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'est'], ['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 'te', 'st'], ['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 'te', 's', 't'], ['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'e', 'st'], ['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'es', 't']]
++->>> for x in range(10):
++-... sp.SampleEncodeAsPieces("This is a test", -1, 0.1)
++-...
++-['\xe2\x96\x81', 'T', 'h', 'i', 's', '\xe2\x96\x81', 'is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'e', 's', 't']
++-['\xe2\x96\x81T', 'h', 'is', '\xe2\x96\x81is', '\xe2\x96\x81', 'a', '\xe2\x96\x81', 't', 'est']
++-['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81', 'a', '\xe2\x96\x81', 't', 'e', 'st']
++-['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'e', 'st']
++-['\xe2\x96\x81This', '\xe2\x96\x81is', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'e', 's', 't']
++-['\xe2\x96\x81T', 'h', 'is', '\xe2\x96\x81', 'i', 's', '\xe2\x96\x81a', '\xe2\x96\x81', 'te', 's', 't']
++-['\xe2\x96\x81This', '\xe2\x96\x81', 'is', '\xe2\x96\x81a', '\xe2\x96\x81', 'te', 's', 't']
++-['\xe2\x96\x81This', '\xe2\x96\x81', 'i', 's', '\xe2\x96\x81a', '\xe2\x96\x81', 't', 'e', 'st']
++-['\xe2\x96\x81This', '\xe2\x96\x81', 'is', '\xe2\x96\x81', 'a', '\xe2\x96\x81', 't', 'e', 'st']
++-['\xe2\x96\x81This', '\xe2\x96\x81', 'i', 's', '\xe2\x96\x81', 'a', '\xe2\x96\x81', 'te', 's', 't']
++->>> sp.DecodeIds([284, 47, 11, 4, 15, 400])
++-'This is a test'
++->>> sp.GetPieceSize()
++-1000
++->>> sp.IdToPiece(2)
++-'</s>'
++->>> sp.PieceToId('</s>')
++-2
++->>> len(sp)
++-1000
++->>> sp['</s>']
++-2
++-```
++-
++-### Model Training (old interface)
++-Training is performed by passing parameters of [spm_train](https://github.com/google/sentencepiece#train-sentencepiece-model) to SentencePieceTrainer.Train() function.
++-
++-```
++->>> import sentencepiece as spm
++->>> spm.SentencePieceTrainer.Train('--input=test/botchan.txt --model_prefix=m --vocab_size=1000')
++-unigram_model_trainer.cc(494) LOG(INFO) Starts training with :
++-input: "test/botchan.txt"
++-model_prefix: "m"
++-model_type: UNIGRAM
++-..snip..
++-unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=0 size=1239 obj=10.4055 num_tokens=36256 num_tokens/piece=29.2623
++-unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=1 size=1239 obj=10.3187 num_tokens=36256 num_tokens/piece=29.2623
++-unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=0 size=1100 obj=10.5285 num_tokens=37633 num_tokens/piece=34.2118
++-unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=1 size=1100 obj=10.4973 num_tokens=37630 num_tokens/piece=34.2091
++-trainer_interface.cc(284) LOG(INFO) Saving model: m.model
++-trainer_interface.cc(293) LOG(INFO) Saving vocabs: m.vocab
++->>>
++-```
++-
++-## Python2/3 String/Unicode compatibility
++-Sentencepiece python wrapper accepts both Unicode string and legacy byte string.
++-The output string type is determined by the input string type.
++-The output type of IdToPiece/DecodeIds methods is *str*, but note that it is a legacy byte string in Python2 and Unicode string in Python3 respectively.
++-
++-* Python2:
++-```
++->>> sp.EncodeAsPieces('吾輩は猫である')
++-['\xe2\x96\x81', '\xe5\x90\xbe', '\xe8\xbc\xa9', '\xe3\x81\xaf', '\xe7\x8c\xab', '\xe3\x81\xa7\xe3\x81\x82\xe3\x82\x8b']
++->>> sp.EncodeAsPieces(u'吾輩は猫である')
++-[u'\u2581', u'\u543e', u'\u8f29', u'\u306f', u'\u732b', u'\u3067\u3042\u308b']
++->>> sp.EncodeAsPieces(u'吾輩は猫である'.encode('utf-8'))
++-['\xe2\x96\x81', '\xe5\x90\xbe', '\xe8\xbc\xa9', '\xe3\x81\xaf', '\xe7\x8c\xab', '\xe3\x81\xa7\xe3\x81\x82\xe3\x82\x8b']
++->>> sp.IdToPiece(10)
++-'\xe3\x81\xab'
++->>> type(sp.IdToPiece(10))
++-<type 'str'>
++-```
++-
++-* Python3:
++-```
++->>> sp.EncodeAsPieces('吾輩は猫である')
++-['▁', '吾', '輩', 'は', '猫', 'である']
++->>> sp.EncodeAsPieces('吾輩は猫である'.encode('utf-8'))
++-[b'\xe2\x96\x81', b'\xe5\x90\xbe', b'\xe8\xbc\xa9', b'\xe3\x81\xaf', b'\xe7\x8c\xab', b'\xe3\x81\xa7\xe3\x81\x82\xe3\x82\x8b']
++->>>
++->>> sp.IdToPiece(10)
++-'に'
++->>> type(sp.IdToPiece(10))
++-<class 'str'>
++-```
++diff --git a/python/src/sentencepiece/__init__.py b/python/src/sentencepiece/__init__.py
++index cf06830..911a2cb 100644
++--- a/python/src/sentencepiece/__init__.py
+++++ b/python/src/sentencepiece/__init__.py
++@@ -635,7 +635,7 @@ class SentencePieceProcessor(object):
++ return _encode(input)
++
++
++- def NBestEncodeAsPieces(self, input, nbest_size=None, **kwargs):
+++ def NBestEncodeAsPieces(self, input, nbest_size=None, **kwargs):
++ return self.NBestEncode(input=input, nbest_size=nbest_size,
++ out_type=str, **kwargs)
++
++@@ -732,6 +732,26 @@ class SentencePieceProcessor(object):
++ return _encode(input)
++
++
+++ def SampleEncodeAndScoreAsPieces(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type=str, **kwargs)
+++
+++
+++ def SampleEncodeAndScoreAsIds(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type=int, **kwargs)
+++
+++
+++ def SampleEncodeAndScoreAsSerializedProto(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type='serialized_proto', **kwargs)
+++
+++
+++ def SampleEncodeAndScoreAsImmutableProto(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type='immutable_proto', **kwargs)
+++
+++
++ def Decode(self, input, out_type=str, num_threads=None):
++ """Decode processed id or token sequences.
++
++diff --git a/python/src/sentencepiece/sentencepiece.i b/python/src/sentencepiece/sentencepiece.i
++index 2ac68a8..fc773e2 100644
++--- a/python/src/sentencepiece/sentencepiece.i
+++++ b/python/src/sentencepiece/sentencepiece.i
++@@ -903,7 +903,7 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return _encode(input)
++
++
++- def NBestEncodeAsPieces(self, input, nbest_size=None, **kwargs):
+++ def NBestEncodeAsPieces(self, input, nbest_size=None, **kwargs):
++ return self.NBestEncode(input=input, nbest_size=nbest_size,
++ out_type=str, **kwargs)
++
++@@ -1000,6 +1000,26 @@ inline void InitNumThreads(const std::vector<T> &ins, int *num_threads) {
++ return _encode(input)
++
++
+++ def SampleEncodeAndScoreAsPieces(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type=str, **kwargs)
+++
+++
+++ def SampleEncodeAndScoreAsIds(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type=int, **kwargs)
+++
+++
+++ def SampleEncodeAndScoreAsSerializedProto(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type='serialized_proto', **kwargs)
+++
+++
+++ def SampleEncodeAndScoreAsImmutableProto(self, input, num_samples=None, alpha=None, **kwargs):
+++ return self.SampleEncodeAndScore(input=input, num_samples=num_samples, alpha=alpha,
+++ out_type='immutable_proto', **kwargs)
+++
+++
++ def Decode(self, input, out_type=str, num_threads=None):
++ """Decode processed id or token sequences.
++
++diff --git a/python/test/sentencepiece_test.py b/python/test/sentencepiece_test.py
++index 92327ac..2b9ad28 100755
++--- a/python/test/sentencepiece_test.py
+++++ b/python/test/sentencepiece_test.py
++@@ -566,7 +566,7 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ for n in sp.decode(results):
++ self.assertEqual(n, text)
++
++- # batch test
+++ # batch test
++ results = sp.nbest_encode([text, text2], nbest_size=10, out_type=out_type)
++ self.assertEqual(
++ results,
++@@ -589,6 +589,19 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ for n in decoded:
++ self.assertEqual(n, text2)
++
+++ self.assertEqual(
+++ sp.nbest_encode(text, nbest_size=10, out_type=str),
+++ sp.nbest_encode_as_pieces(text, nbest_size=10))
+++ self.assertEqual(
+++ sp.nbest_encode(text, nbest_size=10, out_type=int),
+++ sp.nbest_encode_as_ids(text, nbest_size=10))
+++ self.assertEqual(
+++ sp.nbest_encode(text, nbest_size=10, out_type='serialized_proto'),
+++ sp.nbest_encode_as_serialized_proto(text, nbest_size=10))
+++ self.assertEqual(
+++ sp.nbest_encode(text, nbest_size=10, out_type='immutable_proto'),
+++ sp.nbest_encode_as_immutable_proto(text, nbest_size=10))
+++
++ def test_sample_and_score(self):
++ sp = self.sp_
++ text = 'hello world'
++@@ -618,6 +631,11 @@ class TestSentencepieceProcessor(unittest.TestCase):
++ for n in results[1]:
++ self.assertEqual(sp.decode(n[0]), text2)
++
+++ sp.sample_encode_and_score_as_pieces(text, 10)
+++ sp.sample_encode_and_score_as_ids(text, 10)
+++ sp.sample_encode_and_score_as_immutable_proto(text, 10)
+++ sp.sample_encode_and_score_as_serialized_proto(text, 10)
+++
++ def test_valid_range(self):
++ size = self.sp_.piece_size()
++ funcs = [
--- /dev/null
--- /dev/null
++From: Aleksey Morozov <36787333+amrzv@users.noreply.github.com>
++Date: Tue, 9 Aug 2022 15:15:30 +0300
++Subject: Fixed errors in example notebook
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ python/sentencepiece_python_module_example.ipynb | 44 ++++++++++--------------
++ 1 file changed, 19 insertions(+), 25 deletions(-)
++
++diff --git a/python/sentencepiece_python_module_example.ipynb b/python/sentencepiece_python_module_example.ipynb
++index 78464d1..1eb0f9c 100644
++--- a/python/sentencepiece_python_module_example.ipynb
+++++ b/python/sentencepiece_python_module_example.ipynb
++@@ -216,7 +216,7 @@
++ "import tensorflow as tf\n",
++ "\n",
++ "# Assumes that m.model is stored in non-Posix file system.\n",
++- "serialized_model_proto = tf.gfile.GFile('m.model', 'rb').read()\n",
+++ "serialized_model_proto = tf.io.gfile.GFile('m.model', 'rb').read()\n",
++ "\n",
++ "sp = spm.SentencePieceProcessor()\n",
++ "sp.load_from_serialized_proto(serialized_model_proto)\n",
++@@ -265,7 +265,7 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "## Example of user defined symbols\n",
+++ "# Example of user defined symbols\n",
++ "spm.SentencePieceTrainer.train('--input=botchan.txt --model_prefix=m_user --user_defined_symbols=<sep>,<cls> --vocab_size=2000')\n",
++ "\n",
++ "sp_user = spm.SentencePieceProcessor()\n",
++@@ -307,7 +307,7 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "## Example of control symbols\n",
+++ "# Example of control symbols\n",
++ "spm.SentencePieceTrainer.train('--input=botchan.txt --model_prefix=m_ctrl --control_symbols=<sep>,<cls> --vocab_size=2000')\n",
++ "\n",
++ "sp_ctrl = spm.SentencePieceProcessor()\n",
++@@ -564,7 +564,7 @@
++ "spm.SentencePieceTrainer.train('--input=botchan.txt --vocab_size=2000 --model_prefix=m --unk_surface=__UNKNOWN__')\n",
++ "sp = spm.SentencePieceProcessor()\n",
++ "sp.load('m.model')\n",
++- "print(sp.decode_ids([sp.unk_id()])) "
+++ "print(sp.decode_ids([sp.unk_id()]))"
++ ],
++ "execution_count": 0,
++ "outputs": [
++@@ -608,7 +608,7 @@
++ "# There are two hyperparamenters for sampling (nbest_size and inverse temperature). see the paper [kudo18] for detail.\n",
++ "for n in range(10):\n",
++ " print(sp.sample_encode_as_pieces('hello world', -1, 0.1))\n",
++- " \n",
+++ "\n",
++ "for n in range(10):\n",
++ " print(sp.sample_encode_as_ids('hello world', -1, 0.1))"
++ ],
++@@ -858,8 +858,6 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "import sentencepiece as spm\n",
++- "\n",
++ "# NFKC normalization and lower casing.\n",
++ "spm.SentencePieceTrainer.train('--input=botchan.txt --model_prefix=m --vocab_size=2000 --normalization_rule_name=nfkc_cf')\n",
++ "\n",
++@@ -903,11 +901,12 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "def tocode(s): \n",
++- " out = [] \n",
++- " for c in s: \n",
++- " out.append(str(hex(ord(c))).replace('0x', 'U+')) \n",
++- " return ' '.join(out) \n",
+++ "def tocode(s):\n",
+++ " out = []\n",
+++ " for c in s:\n",
+++ " out.append(str(hex(ord(c))).replace('0x', 'U+'))\n",
+++ " return ' '.join(out)\n",
+++ "\n",
++ "\n",
++ "# TSV format: source Unicode code points <tab> target code points\n",
++ "# normalize \"don't => do not, I'm => I am\"\n",
++@@ -923,7 +922,7 @@
++ "# m.model embeds the normalization rule compiled into an FST.\n",
++ "sp.load('m.model')\n",
++ "print(sp.encode_as_pieces(\"I'm busy\")) # normalzied to `I am busy'\n",
++- "print(sp.encode_as_pieces(\"I don't know it.\")) # normalized to 'I do not know it.'\n"
+++ "print(sp.encode_as_pieces(\"I don't know it.\")) # normalized to 'I do not know it.'"
++ ],
++ "execution_count": 0,
++ "outputs": [
++@@ -1029,9 +1028,9 @@
++ " for piece in sp.encode_as_pieces(line):\n",
++ " freq.setdefault(piece, 0)\n",
++ " freq[piece] += 1\n",
++- " \n",
+++ "\n",
++ "# only uses the token appearing more than 1000 times in the training data.\n",
++- "vocabs = list(filter(lambda x : x in freq and freq[x] > 1000, vocabs))\n",
+++ "vocabs = list(filter(lambda x: x in freq and freq[x] > 1000, vocabs))\n",
++ "sp.set_vocabulary(vocabs)\n",
++ "print(sp.encode_as_pieces('this is a test.'))\n",
++ "\n",
++@@ -1133,20 +1132,17 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "freq={}\n",
+++ "freq = {}\n",
++ "with open('botchan.txt', 'r') as f:\n",
++ " for line in f:\n",
++ " line = line.rstrip()\n",
++ " for piece in line.split():\n",
++ " freq.setdefault(piece, 0)\n",
++ " freq[piece] += 1\n",
++- " \n",
+++ "\n",
++ "with open('word_freq_list.tsv', 'w') as f:\n",
++ " for k, v in freq.items():\n",
++ " f.write('%s\\t%d\\n' % (k, v))\n",
++- " \n",
++- "\n",
++- "import sentencepiece as spm\n",
++ "\n",
++ "spm.SentencePieceTrainer.train('--input=word_freq_list.tsv --input_format=tsv --model_prefix=m --vocab_size=2000')\n",
++ "sp = spm.SentencePieceProcessor()\n",
++@@ -1176,7 +1172,7 @@
++ "\n",
++ "Sentencepiece keeps track of byte offset (span) of each token, which is useful for highlighting the token on top of unnormalized text.\n",
++ "\n",
++- "We first need to install protobuf module and sentencepiece_pb2.py as the byte offsets and all other meta data for segementation are encoded in protocol buffer.\n",
+++ "We first need to install protobuf module as the byte offsets and all other meta data for segementation are encoded in protocol buffer.\n",
++ "**encode_as_serialized_proto** method resturns serialized SentencePieceText proto. You can get the deserialized object by calling ParseFromString method.\n",
++ "\n",
++ "The definition of SentencePieceText proto is found [here](https://github.com/google/sentencepiece/blob/3be3f2e11e2bb923c579c6be5e7335809341587f/src/sentencepiece.proto#L23).\n"
++@@ -1194,8 +1190,7 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "!pip install protobuf\n",
++- "!wget https://raw.githubusercontent.com/google/sentencepiece/master/python/sentencepiece_pb2.py"
+++ "!pip install protobuf"
++ ],
++ "execution_count": 0,
++ "outputs": [
++@@ -1233,8 +1228,7 @@
++ },
++ "cell_type": "code",
++ "source": [
++- "import sentencepiece_pb2\n",
++- "import sentencepiece as spm\n",
+++ "from sentencepiece import sentencepiece_pb2\n",
++ "\n",
++ "spm.SentencePieceTrainer.train('--input=botchan.txt --model_prefix=m --vocab_size=2000')\n",
++ "\n",
--- /dev/null
--- /dev/null
++From: Aleksey Morozov <36787333+amrzv@users.noreply.github.com>
++Date: Tue, 9 Aug 2022 15:15:51 +0300
++Subject: Fix dead links
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ README.md | 2 +-
++ doc/experiments.md | 2 +-
++ 2 files changed, 2 insertions(+), 2 deletions(-)
++
++diff --git a/README.md b/README.md
++index 1986047..84e853e 100644
++--- a/README.md
+++++ b/README.md
++@@ -36,7 +36,7 @@ For those unfamiliar with SentencePiece as a software/algorithm, one can read [a
++ |:---|:---:|:---:|:---:|
++ |Supported algorithm|BPE, unigram, char, word|BPE|BPE*|
++ |OSS?|Yes|Yes|Google internal|
++-|Subword regularization|[Yes](#subword-regularization)|No|No|
+++|Subword regularization|[Yes](#subword-regularization-and-bpe-dropout)|No|No|
++ |Python Library (pip)|[Yes](python/README.md)|No|N/A|
++ |C++ Library|[Yes](doc/api.md)|No|N/A|
++ |Pre-segmentation required?|[No](#whitespace-is-treated-as-a-basic-symbol)|Yes|Yes|
++diff --git a/doc/experiments.md b/doc/experiments.md
++index 5a58cd1..e088152 100644
++--- a/doc/experiments.md
+++++ b/doc/experiments.md
++@@ -112,7 +112,7 @@ We have evaluated SentencePiece segmentation with the following configurations.
++ * [KFTT](http://www.phontron.com/kftt/index.html)
++ * [MultiUN](http://opus.lingfil.uu.se/MultiUN.php) (First 5M and next
++ 5k/5k sentences are used for training and development/testing respectively.)
++- * [WMT16](http://www.statmt.org/WMT16/)
+++ * [WMT16](https://www.statmt.org/wmt16/)
++ * In-house: (Used 5M parallel sentences for training)
++
++ **NoPretok** and **WsPretok** do not use any language-dependent resources.
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Sat, 20 Aug 2022 23:34:37 +0900
++Subject: added ShutdownLibrary function to uninitialize global variables
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ src/compile_charsmap_main.cc | 1 +
++ src/error.cc | 3 +++
++ src/init.h | 15 +++++++++++++++
++ src/spm_decode_main.cc | 1 +
++ src/spm_encode_main.cc | 1 +
++ src/spm_export_vocab_main.cc | 6 +++---
++ src/spm_normalize_main.cc | 1 +
++ src/spm_train_main.cc | 1 +
++ src/test_main.cc | 1 +
++ 9 files changed, 27 insertions(+), 3 deletions(-)
++
++diff --git a/src/compile_charsmap_main.cc b/src/compile_charsmap_main.cc
++index 13bf822..da15328 100644
++--- a/src/compile_charsmap_main.cc
+++++ b/src/compile_charsmap_main.cc
++@@ -156,6 +156,7 @@ struct BinaryBlob {
++ } // namespace sentencepiece
++
++ int main(int argc, char **argv) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++
++ const std::vector<std::pair<
++diff --git a/src/error.cc b/src/error.cc
++index 10faa2d..d3792dc 100644
++--- a/src/error.cc
+++++ b/src/error.cc
++@@ -15,6 +15,7 @@
++ #include <cstring>
++
++ #include "common.h"
+++#include "init.h"
++ #include "sentencepiece_processor.h"
++
++ #ifdef _USE_EXTERNAL_ABSL
++@@ -35,6 +36,7 @@ void Abort() {
++ SetTestCounter(2);
++ } else {
++ std::cerr << "Program terminated with an unrecoverable error." << std::endl;
+++ ShutdownLibrary();
++ exit(-1);
++ }
++ }
++@@ -43,6 +45,7 @@ void Exit(int code) {
++ if (GetTestCounter() == 1) {
++ SetTestCounter(2);
++ } else {
+++ ShutdownLibrary();
++ exit(code);
++ }
++ }
++diff --git a/src/init.h b/src/init.h
++index 090a2d9..7c75db2 100644
++--- a/src/init.h
+++++ b/src/init.h
++@@ -18,6 +18,7 @@
++ #include "common.h"
++ #include "third_party/absl/flags/flag.h"
++ #include "third_party/absl/flags/parse.h"
+++#include "third_party/protobuf-lite/google/protobuf/message_lite.h"
++
++ ABSL_DECLARE_FLAG(int32, minloglevel);
++
++@@ -35,6 +36,20 @@ inline void ParseCommandLineFlags(const char *usage, int *argc, char ***argv,
++
++ logging::SetMinLogLevel(absl::GetFlag(FLAGS_minloglevel));
++ }
+++
+++inline void ShutdownLibrary() {
+++ google::protobuf::ShutdownProtobufLibrary();
+++#ifdef HAS_ABSL_CLEANUP_FLAGS
+++ absl::CleanupFlags();
+++#endif
+++}
+++
+++class ScopedResourceDestructor {
+++ public:
+++ ScopedResourceDestructor() {}
+++ ~ScopedResourceDestructor() { ShutdownLibrary(); }
+++};
+++
++ } // namespace sentencepiece
++
++ #endif // INIT_H_
++diff --git a/src/spm_decode_main.cc b/src/spm_decode_main.cc
++index 3382ddc..bc49bd3 100644
++--- a/src/spm_decode_main.cc
+++++ b/src/spm_decode_main.cc
++@@ -34,6 +34,7 @@ ABSL_FLAG(std::string, extra_options, "",
++ "':' separated encoder extra options, e.g., \"reverse:bos:eos\"");
++
++ int main(int argc, char *argv[]) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++ std::vector<std::string> rest_args;
++
++diff --git a/src/spm_encode_main.cc b/src/spm_encode_main.cc
++index b0e508d..2fbb850 100644
++--- a/src/spm_encode_main.cc
+++++ b/src/spm_encode_main.cc
++@@ -51,6 +51,7 @@ ABSL_FLAG(bool, generate_vocabulary, false,
++ "Generates vocabulary file instead of segmentation");
++
++ int main(int argc, char *argv[]) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++ std::vector<std::string> rest_args;
++
++diff --git a/src/spm_export_vocab_main.cc b/src/spm_export_vocab_main.cc
++index b5d93cb..e5b97df 100644
++--- a/src/spm_export_vocab_main.cc
+++++ b/src/spm_export_vocab_main.cc
++@@ -1,11 +1,10 @@
++-
++-
++ // Copyright 2016 Google Inc.
++ //
++ // Licensed under the Apache License, Version 2.0 (the "License");
++ // you may not use this file except in compliance with the License.
++ // You may obtain a copy of the License at
++-// n// http://www.apache.org/licenses/LICENSE-2.0
+++//
+++// http://www.apache.org/licenses/LICENSE-2.0
++ //
++ // Unless required by applicable law or agreed to in writing, software
++ // distributed under the License is distributed on an "AS IS" BASIS,
++@@ -29,6 +28,7 @@ ABSL_FLAG(std::string, output_format, "vocab",
++ "and scores, syms outputs pieces and indices.");
++
++ int main(int argc, char *argv[]) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++
++ sentencepiece::SentencePieceProcessor sp;
++diff --git a/src/spm_normalize_main.cc b/src/spm_normalize_main.cc
++index 96da360..39f3ef9 100644
++--- a/src/spm_normalize_main.cc
+++++ b/src/spm_normalize_main.cc
++@@ -46,6 +46,7 @@ using sentencepiece::normalizer::Builder;
++ using sentencepiece::normalizer::Normalizer;
++
++ int main(int argc, char *argv[]) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++ std::vector<std::string> rest_args;
++
++diff --git a/src/spm_train_main.cc b/src/spm_train_main.cc
++index c34ee02..6ab634d 100644
++--- a/src/spm_train_main.cc
+++++ b/src/spm_train_main.cc
++@@ -157,6 +157,7 @@ ABSL_FLAG(std::uint64_t, differential_privacy_clipping_threshold, 0,
++ " clipping the counts for DP");
++
++ int main(int argc, char *argv[]) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++
++ sentencepiece::TrainerSpec trainer_spec;
++diff --git a/src/test_main.cc b/src/test_main.cc
++index b3170e2..38c978d 100644
++--- a/src/test_main.cc
+++++ b/src/test_main.cc
++@@ -24,6 +24,7 @@ ABSL_FLAG(std::string, test_srcdir, "../data", "Data directory.");
++ ABSL_FLAG(std::string, test_tmpdir, "test_tmp", "Temporary directory.");
++
++ int main(int argc, char **argv) {
+++ sentencepiece::ScopedResourceDestructor cleaner;
++ sentencepiece::ParseCommandLineFlags(argv[0], &argc, &argv, true);
++ sentencepiece::test::RunAllTests();
++ return 0;
--- /dev/null
--- /dev/null
++From: Taku Kudo <taku@google.com>
++Date: Sun, 21 Aug 2022 12:44:31 +0900
++Subject: Fixed the issue of concatinating paths for pkg-config
++
++Signed-off-by: Kentaro Hayashi <kenhys@gmail.com>
++---
++ CMakeLists.txt | 24 ++++++++++++++++++++++++
++ sentencepiece.pc.in | 4 ++--
++ third_party/absl/flags/flag.cc | 20 +++++++++++++++-----
++ third_party/absl/flags/flag.h | 10 ++++++++--
++ 4 files changed, 49 insertions(+), 9 deletions(-)
++
++diff --git a/CMakeLists.txt b/CMakeLists.txt
++index 78379a3..382103b 100644
++--- a/CMakeLists.txt
+++++ b/CMakeLists.txt
++@@ -94,6 +94,30 @@ if (NOT DEFINED CMAKE_INSTALL_INCDIR)
++ set(CMAKE_INSTALL_INCDIR include)
++ endif()
++
+++# SPDX-License-Identifier: (MIT OR CC0-1.0)
+++# Copyright 2020 Jan Tojnar
+++# https://github.com/jtojnar/cmake-snips
+++#
+++# Modelled after Python’s os.path.join
+++# https://docs.python.org/3.7/library/os.path.html#os.path.join
+++# Windows not supported
+++function(join_paths joined_path first_path_segment)
+++ set(temp_path "${first_path_segment}")
+++ foreach(current_segment IN LISTS ARGN)
+++ if(NOT ("${current_segment}" STREQUAL ""))
+++ if(IS_ABSOLUTE "${current_segment}")
+++ set(temp_path "${current_segment}")
+++ else()
+++ set(temp_path "${temp_path}/${current_segment}")
+++ endif()
+++ endif()
+++ endforeach()
+++ set(${joined_path} "${temp_path}" PARENT_SCOPE)
+++endfunction()
+++
+++join_paths(libdir_for_pc_file "\${exec_prefix}" "${CMAKE_INSTALL_LIBDIR}")
+++join_paths(includedir_for_pc_file "\${prefix}" "${CMAKE_INSTALL_INCLUDEDIR}")
+++
++ configure_file("${PROJECT_SOURCE_DIR}/config.h.in" "config.h")
++ configure_file("${PROJECT_SOURCE_DIR}/sentencepiece.pc.in" "sentencepiece.pc" @ONLY)
++
++diff --git a/sentencepiece.pc.in b/sentencepiece.pc.in
++index ac7fef6..6a5ba56 100644
++--- a/sentencepiece.pc.in
+++++ b/sentencepiece.pc.in
++@@ -1,7 +1,7 @@
++ prefix=@prefix@
++ exec_prefix=@exec_prefix@
++-libdir=@libdir@
++-includedir=@includedir@
+++libdir=@libdir_for_pc_file@
+++includedir=@includedir_for_pc_file@
++
++ Name: @PROJECT_NAME@
++ Description: Unsupervised text tokenizer and detokenizer for Neural Network-based text generation.
++diff --git a/third_party/absl/flags/flag.cc b/third_party/absl/flags/flag.cc
++index 8e99c0d..5d6642a 100644
++--- a/third_party/absl/flags/flag.cc
+++++ b/third_party/absl/flags/flag.cc
++@@ -61,8 +61,8 @@ struct FlagFunc {
++
++ namespace {
++
++-using FlagMap = std::map<std::string, FlagFunc *>;
++-using FlagList = std::vector<FlagFunc *>;
+++using FlagMap = std::map<std::string, std::shared_ptr<FlagFunc>>;
+++using FlagList = std::vector<std::shared_ptr<FlagFunc>>;
++
++ FlagMap *GetFlagMap() {
++ static auto *flag_map = new FlagMap;
++@@ -111,7 +111,7 @@ std::string PrintHelp(const char *programname) {
++ os << PACKAGE_STRING << "\n\n";
++ os << "Usage: " << programname << " [options] files\n\n";
++
++- for (const auto *func : *GetFlagList()) {
+++ for (auto func : *GetFlagList()) {
++ os << " --" << func->name << " (" << func->help << ")";
++ os << " type: " << func->type << " default: " << func->default_value
++ << '\n';
++@@ -123,7 +123,7 @@ std::string PrintHelp(const char *programname) {
++ }
++ } // namespace
++
++-void RegisterFlag(const std::string &name, FlagFunc *func) {
+++void RegisterFlag(const std::string &name, std::shared_ptr<FlagFunc> func) {
++ GetFlagList()->emplace_back(func);
++ GetFlagMap()->emplace(name, func);
++ }
++@@ -140,7 +140,7 @@ Flag<T>::Flag(const char *name, const char *type, const char *help,
++ func_->set_value = [this](const std::string &value) {
++ this->set_value_as_str(value);
++ };
++- RegisterFlag(name, func_.get());
+++ RegisterFlag(name, func_);
++ }
++
++ template <typename T>
++@@ -219,4 +219,14 @@ std::vector<char *> ParseCommandLine(int argc, char *argv[]) {
++
++ return output_args;
++ }
+++
+++void CleanupFlags() {
+++ static bool is_shutdown = false;
+++ if (!is_shutdown) {
+++ delete internal::GetFlagList();
+++ delete internal::GetFlagMap();
+++ is_shutdown = true;
+++ }
+++}
+++
++ } // namespace absl
++diff --git a/third_party/absl/flags/flag.h b/third_party/absl/flags/flag.h
++index e540edf..c522358 100644
++--- a/third_party/absl/flags/flag.h
+++++ b/third_party/absl/flags/flag.h
++@@ -24,7 +24,8 @@ namespace absl {
++ namespace internal {
++ struct FlagFunc;
++
++-void RegisterFlag(const std::string &name, FlagFunc *func);
+++void RegisterFlag(const std::string &name, std::shared_ptr<FlagFunc> func);
+++
++ } // namespace internal
++
++ template <typename T>
++@@ -39,7 +40,7 @@ class Flag {
++
++ private:
++ T value_;
++- std::unique_ptr<internal::FlagFunc> func_;
+++ std::shared_ptr<internal::FlagFunc> func_;
++ };
++
++ template <typename T>
++@@ -52,6 +53,11 @@ void SetFlag(Flag<T> *flag, const V &v) {
++ const T value(v);
++ flag->set_value(value);
++ }
+++
+++#define HAS_ABSL_CLEANUP_FLAGS
+++
+++void CleanupFlags();
+++
++ } // namespace absl
++
++ #define ABSL_FLAG(Type, name, defautl_value, help) \
--- /dev/null
--- /dev/null
++From: Kentaro Hayashi <kenhys@xdump.org>
++Date: Wed, 28 Oct 2020 20:55:20 +0900
++Subject: Disable static library explicitly
++
++---
++ src/CMakeLists.txt | 11 +----------
++ 1 file changed, 1 insertion(+), 10 deletions(-)
++
++diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
++index 6cb3922..e4a23ac 100644
++--- a/src/CMakeLists.txt
+++++ b/src/CMakeLists.txt
++@@ -204,12 +204,6 @@ if (SPM_ENABLE_SHARED)
++ add_library(sentencepiece_train SHARED ${SPM_TRAIN_SRCS})
++ endif()
++
++-add_library(sentencepiece-static STATIC ${SPM_SRCS})
++-add_library(sentencepiece_train-static STATIC ${SPM_TRAIN_SRCS})
++-
++-target_link_libraries(sentencepiece-static INTERFACE ${SPM_LIBS})
++-target_link_libraries(sentencepiece_train-static INTERFACE sentencepiece-static ${SPM_LIBS})
++-
++ if (SPM_ENABLE_SHARED)
++ target_link_libraries(sentencepiece ${SPM_LIBS})
++ target_link_libraries(sentencepiece_train ${SPM_LIBS} sentencepiece)
++@@ -220,7 +214,7 @@ if (SPM_ENABLE_SHARED)
++ (${CMAKE_SYSTEM_PROCESSOR} STREQUAL "sh4"))
++ list(APPEND SPM_LIBS "atomic")
++ endif()
++- set(SPM_INSTALLTARGETS sentencepiece sentencepiece_train sentencepiece-static sentencepiece_train-static)
+++ set(SPM_INSTALLTARGETS sentencepiece sentencepiece_train)
++ set_target_properties(sentencepiece sentencepiece_train PROPERTIES SOVERSION 0 VERSION 0.0.0)
++ set_target_properties(sentencepiece PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS YES)
++ set_target_properties(sentencepiece_train PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS YES)
++@@ -237,9 +231,6 @@ else()
++ set(SPM_INSTALLTARGETS sentencepiece-static sentencepiece_train-static)
++ endif()
++
++-set_target_properties(sentencepiece-static PROPERTIES OUTPUT_NAME "sentencepiece")
++-set_target_properties(sentencepiece_train-static PROPERTIES OUTPUT_NAME "sentencepiece_train")
++-
++ if (NOT MSVC)
++ if (SPM_COVERAGE)
++ set(CMAKE_CXX_FLAGS "-O0 -Wall -fPIC -coverage ${CMAKE_CXX_FLAGS}")
--- /dev/null
--- /dev/null
++From: Kentaro Hayashi <kenhys@gmail.com>
++Date: Mon, 21 Nov 2022 22:17:18 +0900
++Subject: Include necessary headers to ensure IS_BIG_ENDIAN is defined
++
++normalizer.h uses IS_BIG_ENDIAN, which is defined in util.h.
++Include util.h here.
++
++Author: Steve Langasek <steve.langasek@ubuntu.com>
++Last-Update: 2022-08-27
++Forwarded: no
++Bug-Debian: https://bugs.debian.org/1017360
++---
++ src/normalizer.h | 1 +
++ 1 file changed, 1 insertion(+)
++
++diff --git a/src/normalizer.h b/src/normalizer.h
++index c79813c..37fdb8a 100644
++--- a/src/normalizer.h
+++++ b/src/normalizer.h
++@@ -22,6 +22,7 @@
++ #include <vector>
++
++ #include "common.h"
+++#include "util.h"
++ #include "sentencepiece_model.pb.h"
++ #include "sentencepiece_processor.h"
++ #include "third_party/absl/strings/string_view.h"
--- /dev/null
--- /dev/null
++0001-update-python-wrapper.patch
++0002-remove-debug-symbols-from-wheel-package.patch
++0003-allow-tab-character-to-be-used-in-user_defined_symbo.patch
++0004-add-test-to-use-tab-as-user-defined-symbols.patch
++0005-Uses-C-17-by-default.patch
++0006-Uses-std-atomic-to-define-global-variable.patch
++0007-Fix-a-typo.patch
++0008-Uses-absl-string_view-as-much-as-possible.patch
++0009-Fixed-build-break.patch
++0010-Added-ImmutableSentencePiece-class.patch
++0011-add-verbose-option.patch
++0012-Supports-ImmutableSentencePieceText-from-python-modu.patch
++0013-Adds-more-unittests.patch
++0014-Adds-SWIGPYTHON-flag.patch
++0015-remove-unused-ifdef-SWIG-macro.patch
++0016-Fixed-test-failure.patch
++0017-Uses-property-in-immutable-proto.patch
++0018-automatically-detect-the-number-of-CPUs-in-batch-pro.patch
++0019-support-slice-in-pieces-nbests-objects.patch
++0020-Updated-the-document.patch
++0021-Fixed-errors-in-example-notebook.patch
++0022-Fix-dead-links.patch
++0023-added-ShutdownLibrary-function-to-uninitialize-globa.patch
++0024-Fixed-the-issue-of-concatinating-paths-for-pkg-confi.patch
++disable-static-library.patch
++support-python-module-in-place.patch
++header-dependencies.patch
--- /dev/null
--- /dev/null
++From: Kentaro Hayashi <kenhys@gmail.com>
++Date: Mon, 21 Nov 2022 22:13:33 +0900
++Subject: Support to build Python module without pkg-config
++
++---
++ python/setup.py | 36 ++++++++++++++++++++----------------
++ 1 file changed, 20 insertions(+), 16 deletions(-)
++
++diff --git a/python/setup.py b/python/setup.py
++index fdf9394..5170d9a 100755
++--- a/python/setup.py
+++++ b/python/setup.py
++@@ -77,25 +77,29 @@ class build_ext(_build_ext):
++ """Override build_extension to run cmake."""
++
++ def build_extension(self, ext):
++- cflags, libs = get_cflags_and_libs('../build/root')
++- if len(libs) == 0:
++- cflags, libs = get_cflags_and_libs('./bundled/root')
++-
++- if len(libs) == 0:
++- if is_sentencepiece_installed():
++- cflags = cflags + run_pkg_config('cflags')
++- libs = run_pkg_config('libs')
++- else:
++- subprocess.check_call(['./build_bundled.sh', __version__])
++- cflags, libs = get_cflags_and_libs('./bundled/root')
+++ # cflags, libs = get_cflags_and_libs('../build/root')
+++ # if len(libs) == 0:
+++ # cflags, libs = get_cflags_and_libs('./bundled/root')
+++
+++ # if len(libs) == 0:
+++ # if is_sentencepiece_installed():
+++ # cflags = cflags + run_pkg_config('cflags')
+++ # libs = run_pkg_config('libs')
+++ # else:
+++ # subprocess.check_call(['./build_bundled.sh', __version__])
+++ # cflags, libs = get_cflags_and_libs('./bundled/root')
++
++ # Fix compile on some versions of Mac OSX
++ # See: https://github.com/neulab/xnmt/issues/199
++- if sys.platform == 'darwin':
++- cflags.append('-mmacosx-version-min=10.9')
++- else:
++- cflags.append('-Wl,-strip-all')
++- libs.append('-Wl,-strip-all')
+++ # if sys.platform == 'darwin':
+++ # cflags.append('-mmacosx-version-min=10.9')
+++ # else:
+++ # cflags.append('-Wl,-strip-all')
+++ # libs.append('-Wl,-strip-all')
+++ cflags = ['-I../src']
+++ cmd = "dpkg-architecture -q DEB_BUILD_GNU_TYPE"
+++ arch = subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split()[0]
+++ libs = ["-L../obj-%s/src" % arch, "-lsentencepiece", "-lsentencepiece_train"]
++ print('## cflags={}'.format(' '.join(cflags)))
++ print('## libs={}'.format(' '.join(libs)))
++ ext.extra_compile_args = cflags
--- /dev/null
--- /dev/null
++usr/lib/python3.*/
--- /dev/null
--- /dev/null
++#!/usr/bin/make -f
++# -*- makefile -*-
++# Sample debian/rules that uses debhelper.
++# This file was originally written by Joey Hess and Craig Small.
++# As a special exception, when this file is copied by dh-make into a
++# dh-make output file, you may use that output file without restriction.
++# This special exception was added by Craig Small in version 0.37 of dh-make.
++
++# Uncomment this to turn on verbose mode.
++#export DH_VERBOSE=1
++export DEB_BUILD_MAINT_OPTIONS = hardening=+all
++DPKG_EXPORT_BUILDFLAGS = 1
++include /usr/share/dpkg/buildflags.mk
++
++ifneq (,$(filter $(DEB_HOST_ARCH), armel mipsel m68k powerpc sh4))
++ export DEB_LDFLAGS_MAINT_APPEND += -Wl,--no-as-needed -latomic -Wl,--as-needed
++endif
++
++%:
++ dh $@ --with python3 --buildsystem=cmake
++
++override_dh_auto_configure:
++ dh_auto_configure --buildsystem=cmake
++ dh_auto_configure --sourcedirectory=python --buildsystem=pybuild
++
++override_dh_auto_build:
++ dh_auto_build --buildsystem=cmake
++ dh_auto_build --sourcedirectory=python --buildsystem=pybuild
++
++override_dh_auto_install: basedir=$(shell pwd)/debian
++override_dh_auto_install:
++ dh_auto_install --buildsystem=cmake
++ dh_auto_install --sourcedirectory=python --buildsystem=pybuild
++
++override_dh_auto_clean:
++ dh_auto_clean --buildsystem=cmake
++ -rm -rf .pybuild
++ -rm -rf .python/sentencepiece.egg-info
++
++# Do no tests.
++override_dh_auto_test:
--- /dev/null
--- /dev/null
++---
++include:
++ - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
++ - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
++
++reprotest:
++ allow_failure: true
--- /dev/null
--- /dev/null
++doc/*.md
--- /dev/null
--- /dev/null
++usr/bin/*
--- /dev/null
--- /dev/null
++<?xml version='1.0' encoding='UTF-8'?>
++<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
++"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
++
++<!--
++
++`xsltproc -''-nonet \
++ -''-param man.charmap.use.subset "0" \
++ -''-param make.year.ranges "1" \
++ -''-param make.single.year.ranges "1" \
++ /usr/share/xml/docbook/stylesheet/nwalsh/manpages/docbook.xsl \
++ manpage.xml'
++
++A manual page <package>.<section> will be generated. You may view the
++manual page with: nroff -man <package>.<section> | less'. A typical entry
++in a Makefile or Makefile.am is:
++
++DB2MAN = /usr/share/sgml/docbook/stylesheet/xsl/nwalsh/manpages/docbook.xsl
++XP = xsltproc -''-nonet -''-param man.charmap.use.subset "0"
++
++manpage.1: manpage.xml
++ $(XP) $(DB2MAN) $<
++
++The xsltproc binary is found in the xsltproc package. The XSL files are in
++docbook-xsl. A description of the parameters you can use can be found in the
++docbook-xsl-doc-* packages. Please remember that if you create the nroff
++version in one of the debian/rules file targets (such as build), you will need
++to include xsltproc and docbook-xsl in your Build-Depends control field.
++Alternatively use the xmlto command/package. That will also automatically
++pull in xsltproc and docbook-xsl.
++
++Notes for using docbook2x: docbook2x-man does not automatically create the
++AUTHOR(S) and COPYRIGHT sections. In this case, please add them manually as
++<refsect1> ... </refsect1>.
++
++To disable the automatic creation of the AUTHOR(S) and COPYRIGHT sections
++read /usr/share/doc/docbook-xsl/doc/manpages/authors.html. This file can be
++found in the docbook-xsl-doc-html package.
++
++Validation can be done using: `xmllint -''-noout -''-valid manpage.xml`
++
++General documentation about man-pages and man-page-formatting:
++man(1), man(7), http://www.tldp.org/HOWTO/Man-Page/
++
++-->
++
++ <!-- Fill in your name for FIRSTNAME and SURNAME. -->
++ <!ENTITY dhfirstname "FIRSTNAME">
++ <!ENTITY dhsurname "SURNAME">
++ <!-- dhusername could also be set to "&firstname; &surname;". -->
++ <!ENTITY dhusername "TSUCHIYA Masatoshi">
++ <!ENTITY dhemail "tsuchiya@namazu.org">
++ <!-- SECTION should be 1-8, maybe w/ subsection other parameters are
++ allowed: see man(7), man(1) and
++ http://www.tldp.org/HOWTO/Man-Page/q2.html. -->
++ <!ENTITY dhsection "SECTION">
++ <!-- TITLE should be something like "User commands" or similar (see
++ http://www.tldp.org/HOWTO/Man-Page/q2.html). -->
++ <!ENTITY dhtitle "sentencepiece User Manual">
++ <!ENTITY dhucpackage "CRFSUITE">
++ <!ENTITY dhpackage "sentencepiece">
++]>
++
++<refentry>
++ <refentryinfo>
++ <title>&dhtitle;</title>
++ <productname>&dhpackage;</productname>
++ <authorgroup>
++ <author>
++ <firstname>&dhfirstname;</firstname>
++ <surname>&dhsurname;</surname>
++ <contrib>Wrote this manpage for the Debian system.</contrib>
++ <address>
++ <email>&dhemail;</email>
++ </address>
++ </author>
++ </authorgroup>
++ <copyright>
++ <year>2007</year>
++ <holder>&dhusername;</holder>
++ </copyright>
++ <legalnotice>
++ <para>This manual page was written for the Debian system
++ (but may be used by others).</para>
++ <para>Permission is granted to copy, distribute and/or modify this
++ document under the terms of the GNU General Public License,
++ Version 2 or (at your option) any later version published by
++ the Free Software Foundation.</para>
++ <para>On Debian systems, the complete text of the GNU General Public
++ License can be found in
++ <filename>/usr/share/common-licenses/GPL</filename>.</para>
++ </legalnotice>
++ </refentryinfo>
++ <refmeta>
++ <refentrytitle>&dhucpackage;</refentrytitle>
++ <manvolnum>&dhsection;</manvolnum>
++ </refmeta>
++ <refnamediv>
++ <refname>&dhpackage;</refname>
++ <refpurpose>program to do something</refpurpose>
++ </refnamediv>
++ <refsynopsisdiv>
++ <cmdsynopsis>
++ <command>&dhpackage;</command>
++ <!-- These are several examples, how syntaxes could look -->
++ <arg choice="plain"><option>-e <replaceable>this</replaceable></option></arg>
++ <arg choice="opt"><option>--example=<parameter>that</parameter></option></arg>
++ <arg choice="opt">
++ <group choice="req">
++ <arg choice="plain"><option>-e</option></arg>
++ <arg choice="plain"><option>--example</option></arg>
++ </group>
++ <replaceable class="option">this</replaceable>
++ </arg>
++ <arg choice="opt">
++ <group choice="req">
++ <arg choice="plain"><option>-e</option></arg>
++ <arg choice="plain"><option>--example</option></arg>
++ </group>
++ <group choice="req">
++ <arg choice="plain"><replaceable>this</replaceable></arg>
++ <arg choice="plain"><replaceable>that</replaceable></arg>
++ </group>
++ </arg>
++ </cmdsynopsis>
++ <cmdsynopsis>
++ <command>&dhpackage;</command>
++ <!-- Normally the help and version options make the programs stop
++ right after outputting the requested information. -->
++ <group choice="opt">
++ <arg choice="plain">
++ <group choice="req">
++ <arg choice="plain"><option>-h</option></arg>
++ <arg choice="plain"><option>--help</option></arg>
++ </group>
++ </arg>
++ <arg choice="plain">
++ <group choice="req">
++ <arg choice="plain"><option>-v</option></arg>
++ <arg choice="plain"><option>--version</option></arg>
++ </group>
++ </arg>
++ </group>
++ </cmdsynopsis>
++ </refsynopsisdiv>
++ <refsect1 id="description">
++ <title>DESCRIPTION</title>
++ <para>This manual page documents briefly the
++ <command>&dhpackage;</command> and <command>bar</command>
++ commands.</para>
++ <para>This manual page was written for the Debian distribution
++ because the original program does not have a manual page.
++ Instead, it has documentation in the GNU <citerefentry>
++ <refentrytitle>info</refentrytitle>
++ <manvolnum>1</manvolnum>
++ </citerefentry> format; see below.</para>
++ <para><command>&dhpackage;</command> is a program that...</para>
++ </refsect1>
++ <refsect1 id="options">
++ <title>OPTIONS</title>
++ <para>The program follows the usual GNU command line syntax,
++ with long options starting with two dashes (`-'). A summary of
++ options is included below. For a complete description, see the
++ <citerefentry>
++ <refentrytitle>info</refentrytitle>
++ <manvolnum>1</manvolnum>
++ </citerefentry> files.</para>
++ <variablelist>
++ <!-- Use the variablelist.term.separator and the
++ variablelist.term.break.after parameters to
++ control the term elements. -->
++ <varlistentry>
++ <term><option>-e <replaceable>this</replaceable></option></term>
++ <term><option>--example=<replaceable>that</replaceable></option></term>
++ <listitem>
++ <para>Does this and that.</para>
++ </listitem>
++ </varlistentry>
++ <varlistentry>
++ <term><option>-h</option></term>
++ <term><option>--help</option></term>
++ <listitem>
++ <para>Show summary of options.</para>
++ </listitem>
++ </varlistentry>
++ <varlistentry>
++ <term><option>-v</option></term>
++ <term><option>--version</option></term>
++ <listitem>
++ <para>Show version of program.</para>
++ </listitem>
++ </varlistentry>
++ </variablelist>
++ </refsect1>
++ <refsect1 id="files">
++ <title>FILES</title>
++ <variablelist>
++ <varlistentry>
++ <term><filename>/etc/foo.conf</filename></term>
++ <listitem>
++ <para>The system-wide configuration file to control the
++ behaviour of <application>&dhpackage;</application>. See
++ <citerefentry>
++ <refentrytitle>foo.conf</refentrytitle>
++ <manvolnum>5</manvolnum>
++ </citerefentry> for further details.</para>
++ </listitem>
++ </varlistentry>
++ <varlistentry>
++ <term><filename>${HOME}/.foo.conf</filename></term>
++ <listitem>
++ <para>The per-user configuration file to control the
++ behaviour of <application>&dhpackage;</application>. See
++ <citerefentry>
++ <refentrytitle>foo.conf</refentrytitle>
++ <manvolnum>5</manvolnum>
++ </citerefentry> for further details.</para>
++ </listitem>
++ </varlistentry>
++ </variablelist>
++ </refsect1>
++ <refsect1 id="environment">
++ <title>ENVIONMENT</title>
++ <variablelist>
++ <varlistentry>
++ <term><envar>FOO_CONF</envar></term>
++ <listitem>
++ <para>If used, the defined file is used as configuration
++ file (see also <xref linkend="files"/>).</para>
++ </listitem>
++ </varlistentry>
++ </variablelist>
++ </refsect1>
++ <refsect1 id="diagnostics">
++ <title>DIAGNOSTICS</title>
++ <para>The following diagnostics may be issued
++ on <filename class="devicefile">stderr</filename>:</para>
++ <variablelist>
++ <varlistentry>
++ <term><errortext>Bad configuration file. Exiting.</errortext></term>
++ <listitem>
++ <para>The configuration file seems to contain a broken configuration
++ line. Use the <option>--verbose</option> option, to get more info.
++ </para>
++ </listitem>
++ </varlistentry>
++ </variablelist>
++ <para><command>&dhpackage;</command> provides some return codes, that can
++ be used in scripts:</para>
++ <segmentedlist>
++ <segtitle>Code</segtitle>
++ <segtitle>Diagnostic</segtitle>
++ <seglistitem>
++ <seg><errorcode>0</errorcode></seg>
++ <seg>Program exited successfully.</seg>
++ </seglistitem>
++ <seglistitem>
++ <seg><errorcode>1</errorcode></seg>
++ <seg>The configuration file seems to be broken.</seg>
++ </seglistitem>
++ </segmentedlist>
++ </refsect1>
++ <refsect1 id="bugs">
++ <!-- Or use this section to tell about upstream BTS. -->
++ <title>BUGS</title>
++ <para>The program is currently limited to only work
++ with the <package>foobar</package> library.</para>
++ <para>The upstreams <acronym>BTS</acronym> can be found
++ at <ulink url="http://bugzilla.foo.tld"/>.</para>
++ </refsect1>
++ <refsect1 id="see_also">
++ <title>SEE ALSO</title>
++ <!-- In alpabetical order. -->
++ <para><citerefentry>
++ <refentrytitle>bar</refentrytitle>
++ <manvolnum>1</manvolnum>
++ </citerefentry>, <citerefentry>
++ <refentrytitle>baz</refentrytitle>
++ <manvolnum>1</manvolnum>
++ </citerefentry>, <citerefentry>
++ <refentrytitle>foo.conf</refentrytitle>
++ <manvolnum>5</manvolnum>
++ </citerefentry></para>
++ <para>The programs are documented fully by <citetitle>The Rise and
++ Fall of a Fooish Bar</citetitle> available via the <citerefentry>
++ <refentrytitle>info</refentrytitle>
++ <manvolnum>1</manvolnum>
++ </citerefentry> system.</para>
++ </refsect1>
++</refentry>
++
--- /dev/null
--- /dev/null
++3.0 (quilt)
--- /dev/null
--- /dev/null
++version=4
++opts="filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%sentencepiece-$1-Source.tar.xz%" \
++ https://github.com/google/sentencepiece/tags \
++ (?:.*?/)?v(\d[\d.]*)\.tar\.gz debian uupdate
projects / sentencepiece.git / commitdiff