Commit ba362917 authored by krb's avatar krb Committed by Commit bot

[spellcheck] Updated Hunspell to 1.6.0

To pick up our upstreamed optimization for suggestions, and fuzzing bug fixes.

The license requirements were approved, by having README.chromium
specify which license it was using.

BUG=673424

Review-Url: https://codereview.chromium.org/2587363003
Cr-Commit-Position: refs/heads/master@{#443234}
parent 8733eb80
This diff is collapsed.
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.
"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.
An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.
A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".
The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.
The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:
a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license
document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:
a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.
c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.
This diff is collapsed.
About Hunspell
--------------
==============
NOTICE: Verison 2 is in the works. For contributing see
[version 2 specification][v2spec] and the folder `src/hunspell2`.
[v2spec]: https://github.com/hunspell/hunspell/wiki/Version-2-Specification
Hunspell is a spell checker and morphological analyzer library and program
designed for languages with rich morphology and complex word compounding or
character encoding. Hunspell interfaces: Ispell-like terminal interface
using Curses library, Ispell pipe interface, OpenOffice.org UNO module.
using Curses library, Ispell pipe interface, C++ class and C functions.
Hunspell's code base comes from the OpenOffice.org MySpell
(http://lingucomponent.openoffice.org/MySpell-3.zip). See README.MYSPELL,
......@@ -13,31 +18,27 @@ Hunspell is designed to eventually replace Myspell in OpenOffice.org.
Main features of Hunspell spell checker and morphological analyzer:
- Unicode support (affix rules work only with the first 65535 Unicode characters)
- Unicode support (affix rules work only with the first 65535 Unicode
characters)
- Morphological analysis (in custom item and arrangement style) and stemming
- Max. 65535 affix classes and twofold affix stripping (for agglutinative
languages, like Azeri, Basque, Estonian, Finnish, Hungarian, Turkish, etc.)
- Support complex compoundings (for example, Hungarian and German)
- Support language specific features (for example, special casing of
Azeri and Turkish dotted i, or German sharp s)
- Handle conditional affixes, circumfixes, fogemorphemes,
forbidden words, pseudoroots and homonyms.
- Free software (LGPL, GPL, MPL tri-license)
- Free software. Versions 1.x are licenced under LGPL, GPL, MPL tri-license.
Version 2 is licenced only under GNU LGPL.
Compiling on Unix/Linux and others
----------------------------------
==================================
autoreconf -vfi
./configure
make
make install #if neccesary prefix with sudo
ldconfig #not needed on windows, on linux sudo may be needed
autoreconf -vfi
./configure
make
make install #if neccesary prefix with sudo
ldconfig #not needed on windows, on linux sudo may be needed
For dictionary development, use the --with-warnings option of configure.
......@@ -45,128 +46,130 @@ For interactive user interface of Hunspell executable, use the --with-ui option.
The developer packages you need to compile Hunspell's interface:
autoconf automake autopoint libtool g++
autoconf automake autopoint libtool g++
optional developer packages:
ncurses (need for --with-ui), eg. libncursesw5 for UTF-8
readline (for fancy input line editing,
- ncurses (need for --with-ui), eg. libncursesw5 for UTF-8
- readline (for fancy input line editing,
configure parameter: --with-readline)
locale and gettext (but you can also use the
- locale and gettext (but you can also use the
--with-included-gettext configure parameter)
Compiling on Windows
--------------------
====================
1. Compiling with Visual Studio
Download the free Visual Studio Community Edition of Microsoft, open the
file hunspell/src/win_api/Hunspell.sln. Select the appropirate build (Debug,
Release, Win32, x64) and press Build.
2. Compiling with Mingw64 and MSYS2
1. Compiling with Mingw64 and MSYS2
-----------------------------------
Download Msys2, update everything and install the following packages:
pacman -S base-devel mingw-w64-x86_64-toolchain mingw-w64-x86_64-libtool
pacman -S base-devel mingw-w64-x86_64-toolchain mingw-w64-x86_64-libtool
Open Mingw-w64 Win64 prompt and compile the same way as on Linux, see above.
3. Compiling in Cygwin environment
2. Compiling in Cygwin environment
----------------------------------
Download and install Cygwin environment for Windows with the following
extra packages:
make
automake
autoconf
gcc-g++ development package
ncurses, readline (for user interface)
iconv (character conversion)
- make
- automake
- autoconf
- gcc-g++ development package
- ncurses, readline (for user interface)
- iconv (character conversion)
3.1. Cygwin1.dll dependent compiling
###3.1. Cygwin1.dll dependent compiling
Same as on Linux.
Testing
-------
========
Testing Hunspell (see tests in tests/ subdirectory):
make check
make check
or with Valgrind debugger:
make check
VALGRIND=[Valgrind_tool] make check
make check
VALGRIND=[Valgrind_tool] make check
For example:
make check
VALGRIND=memcheck make check
make check
VALGRIND=memcheck make check
Documentation
-------------
=============
features and dictionary format:
man 5 hunspell
man hunspell
hunspell -h
man 5 hunspell
man hunspell
hunspell -h
http://hunspell.github.io/
Usage
-----
The src/tools dictionary contains ten executables after compiling
(or some of them are in the src/win_api):
affixcompress: dictionary generation from large (millions of words) vocabularies
analyze: example of spell checking, stemming and morphological analysis
chmorph: example of automatic morphological generation and conversion
example: example of spell checking and suggestion
hunspell: main program for spell checking and others (see manual)
hunzip: decompressor of hzip format
hzip: compressor of hzip format
makealias: alias compression (Hunspell only, not back compatible with MySpell)
munch: dictionary generation from vocabularies (it needs an affix file, too).
unmunch: list all recognized words of a MySpell dictionary
wordforms: word generation (Hunspell version of unmunch)
=====
The src/tools dictionary contains ten executables after compiling:
- affixcompress: dictionary generation from large (millions of words)
vocabularies
- analyze: example of spell checking, stemming and morphological analysis
- chmorph: example of automatic morphological generation and conversion
- example: example of spell checking and suggestion
- hunspell: main program for spell checking and others (see manual)
- hunzip: decompressor of hzip format
- hzip: compressor of hzip format
- makealias: alias compression (Hunspell only, not back compatible with MySpell)
- munch: dictionary generation from vocabularies (it needs an affix file, too).
- unmunch: list all recognized words of a MySpell dictionary
- wordforms: word generation (Hunspell version of unmunch)
After compiling and installing (see INSTALL) you can
run the Hunspell spell checker (compiled with user interface)
with a Hunspell or Myspell dictionary:
hunspell -d en_US text.txt
hunspell -d en_US text.txt
or without interface:
hunspell
hunspell -d en_UK -l <text.txt
hunspell
hunspell -d en_UK -l <text.txt
Dictionaries consist of an affix and dictionary file, see tests/
or http://wiki.services.openoffice.org/wiki/Dictionaries.
Using Hunspell library with GCC
-------------------------------
===============================
Including in your program:
#include <hunspell.hxx>
#include <hunspell.hxx>
Linking with Hunspell static library:
g++ -lhunspell example.cxx
g++ -lhunspell example.cxx
Dictionaries
------------
Myspell & Hunspell dictionaries:
http://extensions.libreoffice.org
http://cgit.freedesktop.org/libreoffice/dictionaries
http://extensions.openoffice.org
http://wiki.services.openoffice.org/wiki/Dictionaries
- http://extensions.libreoffice.org
- http://cgit.freedesktop.org/libreoffice/dictionaries
- http://extensions.openoffice.org
- http://wiki.services.openoffice.org/wiki/Dictionaries
Aspell dictionaries (need some conversion):
ftp://ftp.gnu.org/gnu/aspell/dict
- ftp://ftp.gnu.org/gnu/aspell/dict
Conversion steps: see relevant feature request at http://hunspell.github.io/ .
László Németh
......
Name: hunspell
URL: http://hunspell.sourceforge.net/
Version: 1.5.4
Version: 1.6.0
License: MPL 1.1/GPL 2.0/LGPL 2.1
License File: COPYING
License File: COPYING.MPL
Security Critical: yes
Description:
This is a partial copy of Hunspell 1.5.4 with the following changes:
* Remove '#include "config.h"' from src/hunspell/hunspell.hxx
This is a partial copy of Hunspell 1.6.0 with the following changes:
* Move README.md onto the README symlink.
* Change src/hunspell/filemgr.hxx and src/hunspell/filemgr.cxx to use
LineIterator.
* Add ScopedHashEntry, which creates temporary hentry objects, to
......@@ -27,7 +27,7 @@ steps, or simply run update_google_patch.sh from the commandline.
1) Checkout hunspell:
$ git clone https://github.com/hunspell/hunspell.git
$ cd hunspell
$ git checkout v1.5.4
$ git checkout v1.6.0
2) Apply the existing patch:
$ patch -p0 -i ~/src/third_party/hunspell/google.patch
3) Make your new changes inside the CVS hunspell directory.
......
This diff is collapsed.
Hunspell spell checker and morphological analyser library
Documentation, tests, examples: http://hunspell.github.io/
Author of Hunspell:
László Németh (nemethl (at) gyorsposta.hu)
Hunspell based on OpenOffice.org's Myspell. MySpell's author:
Kevin Hendricks (kevin.hendricks (at) sympatico.ca)
License: GPL 2.0/LGPL 2.1/MPL 1.1 tri-license
The contents of this library may be used under the terms of
the GNU General Public License Version 2 or later (the "GPL"), or
the GNU Lesser General Public License Version 2.1 or later (the "LGPL",
see http://gnu.org/copyleft/lesser.html) or the Mozilla Public License
Version 1.1 or later (the "MPL", see http://mozilla.org/MPL/MPL-1.1.html).
Software distributed under these licenses is distributed on an "AS IS" basis,
WITHOUT WARRANTY OF ANY KIND, either express or implied. See the licences
for the specific language governing rights and limitations under the licenses.
......@@ -2126,7 +2126,7 @@ struct hentry* AffixMgr::compound_check(const std::string& word,
wordnum = oldwordnum2;
// perhaps second word is a compound word (recursive call)
if (wordnum < maxwordnum) {
if (wordnum + 2 < maxwordnum) {
rv = compound_check(st.substr(i), wordnum + 1,
numsyllable, maxwordnum, wnum + 1, words, rwords, 0,
is_sug, info);
......@@ -2699,7 +2699,7 @@ int AffixMgr::compound_check_morph(const char* word,
wordnum = oldwordnum2;
// perhaps second word is a compound word (recursive call)
if ((wordnum < maxwordnum) && (ok == 0)) {
if ((wordnum + 2 < maxwordnum) && (ok == 0)) {
compound_check_morph((word + i), strlen(word + i), wordnum + 1,
numsyllable, maxwordnum, wnum + 1, words, rwords, 0,
result, &presult);
......
......@@ -87,9 +87,12 @@
#define MAXSUGGESTION 15
#define MAXSHARPS 5
#define MAXWORDLEN 176
#if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
#ifndef MAXWORDLEN
#define MAXWORDLEN 100
#endif
#if defined __GNUC__ && (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1))
# define H_DEPRECATED __attribute__((__deprecated__))
#elif defined(_MSC_VER) && (_MSC_VER >= 1300)
# define H_DEPRECATED __declspec(deprecated)
......
/* ***** BEGIN LICENSE BLOCK *****
* Version: MPL 1.1/GPL 2.0/LGPL 2.1
*
* The contents of this file are subject to the Mozilla Public License Version
* 1.1 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
* http://www.mozilla.org/MPL/
*
* Software distributed under the License is distributed on an "AS IS" basis,
* WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
* for the specific language governing rights and limitations under the
* License.
*
* The Original Code is Hunspell, based on MySpell.
*
* The Initial Developers of the Original Code are
* Kevin Hendricks (MySpell) and Laszlo Nemeth (Hunspell).
* Portions created by the Initial Developers are Copyright (C) 2002-2005
* the Initial Developers. All Rights Reserved.
*
* Contributor(s):
* David Einstein
* Davide Prina
* Giuseppe Modugno
* Gianluca Turconi
* Simon Brouwer
* Noll Janos
* Biro Arpad
* Goldman Eleonora
* Sarlos Tamas
* Bencsath Boldizsar
* Halacsy Peter
* Dvornik Laszlo
* Gefferth Andras
* Nagy Viktor
* Varga Daniel
* Chris Halls
* Rene Engelhard
* Bram Moolenaar
* Dafydd Jones
* Harri Pitkanen
* Andras Timar
* Tor Lillqvist
*
* Alternatively, the contents of this file may be used under the terms of
* either the GNU General Public License Version 2 or later (the "GPL"), or
* the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
* in which case the provisions of the GPL or the LGPL are applicable instead
* of those above. If you wish to allow use of your version of this file only
* under the terms of either the GPL or the LGPL, and not to allow others to
* use your version of this file under the terms of the MPL, indicate your
* decision by deleting the provisions above and replace them with the notice
* and other provisions required by the GPL or the LGPL. If you do not delete
* the provisions above, a recipient may use your version of this file under
* the terms of any one of the MPL, the GPL or the LGPL.
*
* ***** END LICENSE BLOCK ***** */
#ifndef MOZILLA_CLIENT
# include "config.h"
#endif
/*
* Copyright 2002 Kevin B. Hendricks, Stratford, Ontario, Canada
* And Contributors. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* 3. All modifications to the source code must be clearly marked as
* such. Binary redistributions based on modified source code
* must be clearly marked as modified versions in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY KEVIN B. HENDRICKS AND CONTRIBUTORS
* ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
* KEVIN B. HENDRICKS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
*
* NOTE: A special thanks and credit goes to Geoff Kuenning
* the creator of ispell. MySpell's affix algorithms were
* based on those of ispell which should be noted is
* copyright Geoff Kuenning et.al. and now available
* under a BSD style license. For more information on ispell
* and affix compression in general, please see:
* http://www.cs.ucla.edu/ficus-members/geoff/ispell.html
* (the home page for ispell)
*
* An almost complete rewrite of MySpell for use by
* the Mozilla project has been developed by David Einstein
* (Deinst@world.std.com). David and I are now
* working on parallel development tracks to help
* our respective projects (Mozilla and OpenOffice.org
* and we will maintain full affix file and dictionary
* file compatibility and work on merging our versions
* of MySpell back into a single tree. David has been
* a significant help in improving MySpell.
*
* Special thanks also go to La'szlo' Ne'meth
* <nemethl@gyorsposta.hu> who is the author of the
* Hungarian dictionary and who developed and contributed
* the code to support compound words in MySpell
* and fixed numerous problems with the encoding
* case conversion tables.
*
*/
......@@ -181,8 +181,12 @@ class SuggestMgr {
const std::vector<mapentry>&,
int*,
clock_t*);
int ngram(int n, const std::vector<w_char>& su1,
const std::vector<w_char>& su2, int opt);
int ngram(int n, const std::string& s1, const std::string& s2, int opt);
int mystrlen(const char* word);
int leftcommonsubstring(const std::vector<w_char>& su1,
const std::vector<w_char>& su2);
int leftcommonsubstring(const char* s1, const char* s2);
int commoncharacterpositions(const char* s1, const char* s2, int* is_swap);
void bubblesort(char** rwd, char** rwd2, int* rsc, int n);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment