Skip to content

GitLab

T
tangled
Commit 591102e7 authored by Chris Mumford's avatar Chris Mumford Committed by Commit Bot
Browse files
Options

[sqlite] Switch to external SQLite repository. 

This change shifts from building the SQLite source in Chromium
to building the SQLite source which now resides in its own
repository.

Motivation: now that SQLite has an official Git mirror we can
switch from manually maintaining a set of patches to having
a Chromium specific set of branches (master and releases).
This simplifies the maintenance task and this repository.

Bug: 945204
Change-Id: I83febc92d8968a75be69aeb18b352561b8817f0a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2013741Reviewed-by: default avatarMikhail Khokhlov <khokhlov@google.com>
Reviewed-by: default avatarDarwin Huang <huangdarwin@chromium.org>
Reviewed-by: default avatarVictor Costan <pwnall@chromium.org>
Commit-Queue: Chris Mumford <cmumford@google.com>
Cr-Commit-Position: refs/heads/master@{#750587}
parent e27a554c
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1004 additions and 607 deletions
DEPS
View file @ 591102e7
......@@ -1390,6 +1390,9 @@ deps = {
'src/third_party/snappy/src':
Var('chromium_git') + '/external/github.com/google/snappy.git' + '@' + 'f5acee902c4d2110f671455460172cb6d3bf5b73',
'src/third_party/sqlite/src':
Var('chromium_git') + '/chromium/deps/sqlite.git' + '@' + '3d3ac5e6f58a5bc28e96e621909b594e7f950e53',
'src/third_party/sqlite4java': {
'packages': [
{
......
third_party/sqlite/BUILD.gn
View file @ 591102e7
......@@ -6,6 +6,9 @@ import("//build/config/dcheck_always_on.gni")
import("//build/config/sanitizers/sanitizers.gni")
import("//testing/libfuzzer/fuzzer_test.gni")
import("//third_party/protobuf/proto_library.gni")
import("//third_party/sqlite/sqlite_chromium_configuration_flags.gni")
import("//third_party/sqlite/sqlite_common_configuration_flags.gni")
import("//third_party/sqlite/sqlite_dev_configuration_flags.gni")
# Compile-time options passed to SQLite.
#
......@@ -19,94 +22,17 @@ import("//third_party/protobuf/proto_library.gni")
# The vast majority of the macros here are documented at
# https://www.sqlite.org/compile.html
config("common_sqlite3_compile_options") {
defines = [
# Start with the configuration flags which are also baked into the SQLite
# source when the amalgamation is generated. These are the same for all
# platforms.
defines = sqlite_common_configuration_flags
# The flags not baked in, but common for all platforms.
defines += [
# Skip writing transaction rollback journals on f2fs.
# f2fs tends to be used on Android, and may be used on ChromeOS.
"SQLITE_ENABLE_BATCH_ATOMIC_WRITE",
"SQLITE_ENABLE_FTS3",
# New unicode61 tokenizer with built-in tables.
"SQLITE_DISABLE_FTS3_UNICODE",
# Chrome does not enable fts4, disable extra code.
"SQLITE_DISABLE_FTS4_DEFERRED",
"SQLITE_ENABLE_ICU",
# Defaults the secure_delete pragma to 1.
#
# This causes SQLite to overwrite all deleted information with zeroes,
# trading additional I/O for better privacy guarantees.
"SQLITE_SECURE_DELETE",
# TODO(pwnall): SQLite adds mutexes to protect structures which cross
# threads. In theory Chrome should be able to turn this to "2" which
# should give a slight speed boost. "2" is safe as long as a single
# connection is not used by more than one thread at a time.
"SQLITE_THREADSAFE=1",
# SQLite can spawn threads to sort in parallel if configured
# appropriately. Chrome doesn't configure SQLite for that, and would
# prefer to control distribution to worker threads.
"SQLITE_MAX_WORKER_THREADS=0",
# Allow 256MB mmap footprint per connection. Should not be too open-ended
# as that could cause memory fragmentation. 50MB encompasses the 99th
# percentile of Chrome databases in the wild.
# TODO(pwnall): A 64-bit-specific value could be 1G or more.
# TODO(pwnall): Figure out if exceeding this is costly.
"SQLITE_MAX_MMAP_SIZE=268435456",
# The default POSIX permissions for a newly created SQLite database.
#
# If unspecified, this defaults to 0644. All the data stored by Chrome is
# private, so our databases use stricter settings.
"SQLITE_DEFAULT_FILE_PERMISSIONS=0600",
# Needed by the SQL MemoryDumpProvider.
#
# Setting this to 1 is needed to collect the information reported by
# sqlite3_status64(SQLITE_STATUS_MEMORY_USED). Without this setting, the API
# still exists, but does not work as promised.
"SQLITE_DEFAULT_MEMSTATUS=1",
# Must match sql::Database::kDefaultPageSize.
"SQLITE_DEFAULT_PAGE_SIZE=4096",
# By default SQLite pre-allocates 100 pages of pcache data, which will not
# be released until the handle is closed. This is contrary to Chrome's
# memory-usage goals.
"SQLITE_DEFAULT_PCACHE_INITSZ=0",
# Some defines can affect the amalgamation. Those should be added to
# scripts/generate_amalgamation.sh, and the amalgamation re-generated.
# Usually this involves disabling features which include keywords or
# syntax, for instance SQLITE_OMIT_VIRTUALTABLE omits the virtual table
# syntax entirely. Missing an item usually results in syntax working but
# execution failing. Review:
# patched/src/parse.py
# patched/tool/mkkeywordhash.c
# The flags below are recommended in the SQLite documentation.
"SQLITE_LIKE_DOESNT_MATCH_BLOBS",
"SQLITE_OMIT_DEPRECATED",
"SQLITE_OMIT_PROGRESS_CALLBACK",
"SQLITE_OMIT_SHARED_CACHE",
"SQLITE_USE_ALLOCA",
# Chrome does not use sqlite3_column_decltype().
"SQLITE_OMIT_DECLTYPE",
# Chrome does not use sqlite3_{enable_}load_extension().
# Asides from giving us fairly minor code savings, this option disables code
# that breaks our method for renaming SQLite's exported symbols. Last,
# there's a tiny security benefit to knowing that WebSQL can't possibly
# reach extension loading code.
"SQLITE_OMIT_LOAD_EXTENSION",
# Uses isnan() in the C99 standard library.
"SQLITE_HAVE_ISNAN",
# Forces SQLite to only store temporary data in memory.
#
# This is the only supported setting on Android. Using it everywhere removes
......@@ -179,89 +105,18 @@ config("common_sqlite3_compile_options") {
}
}
# These options remove SQLite features that we don't need in Chrome.
# Note that some defines affect amalgamation. Please make sure that this
# list of options is in sync with build flags for Chromium in
# scripts/generate_amalgamation.sh.
# These options add SQLite features that we need in Chrome.
config("chromium_sqlite3_compile_options") {
configs = [ ":common_sqlite3_compile_options" ]
defines = [
# Chrome doesn't use the ANALYZE SQLite extension.
#
# ANALYZE [1] is non-standard, and currently performs a table scan to
# update statistics used by the query planner. Chrome uses straightforward
# database schemas which do not require the level of fine tuning provided
# by ANALYZE, and we generally cannot afford the I/O cost of the required
# table scans.
#
# [1] https://www.sqlite.org/lang_analyze.html
"SQLITE_OMIT_ANALYZE",
# Chrome initializes SQLite manually in //sql/connection.cc.
"SQLITE_OMIT_AUTOINIT",
# Chrome calls sqlite3_reset() correctly to reset prepared statements.
"SQLITE_OMIT_AUTORESET",
# Chromium does not use sqlite3_{get,free}_table().
# Chrome doesn't use sqlite3_compileoption_{used,get}().
"SQLITE_OMIT_COMPILEOPTION_DIAGS",
# Chrome doesn't ship the SQLite shell, so command auto-completion is not
# needed. Chrome developers who build the SQLite shell living at
# //third_party/sqlite:sqlite_shell for diagnostic purposes will have to
# live without auto-completion.
"SQLITE_OMIT_COMPLETE",
# EXPLAIN's output is not stable across releases [1], so it should not be
# used in Chrome. Skipping the EXPLAIN machinery also results in
# non-trivial binary savings.
#
# [1] https://www.sqlite.org/lang_explain.html
"SQLITE_OMIT_EXPLAIN",
# Chrome does not use sqlite3_{get,free}_table().
"SQLITE_OMIT_GET_TABLE",
# Chrome already depends on malloc being very efficient, so we disable
# SQLite's arena allocator.
"SQLITE_DEFAULT_LOOKASIDE=0,0",
"SQLITE_OMIT_LOOKASIDE",
# Chrome doesn't use TCL variables.
"SQLITE_OMIT_TCL_VARIABLE",
# The REINDEX statemnt is only useful if a collation sequence's definition
# changes [1]. Chrome never defines its own collation sequences [2, 3], so
# it never needs to call REINDEX.
#
# [1] https://www.sqlite.org/lang_reindex.html
# [2] https://www.sqlite.org/datatype3.html#collating_sequences
# [3] https://www.sqlite.org/c3ref/create_collation.html
"SQLITE_OMIT_REINDEX",
# Chrome doesn't use sqlite3_{profile,trace}().
"SQLITE_OMIT_TRACE",
# Chrome doesn't use UPSERT.
"SQLITE_OMIT_UPSERT",
# Chrome doesn't use window functions in SQL.
"SQLITE_OMIT_WINDOWFUNC",
]
defines = sqlite_chromium_configuration_flags
}
# These options add SQLite features that we need in developer tools.
# Note that some defines affect amalgamation. Please make sure that this
# list of options is in sync with build flags for dev version in
# scripts/generate_amalgamation.sh.
config("dev_sqlite3_compile_options") {
configs = [ ":common_sqlite3_compile_options" ]
defines = [
"SQLITE_ENABLE_EXPLAIN_COMMENTS",
]
defines = sqlite_dev_configuration_flags
}
config("sqlite_warnings") {
......@@ -313,14 +168,14 @@ component("chromium_sqlite3") {
public = [ "sqlite3.h" ]
sources = [
"amalgamation/sqlite3.h",
"sqlite3_shim.c",
"src/amalgamation/sqlite3.h",
]
inputs = [
# This file is #included into sqlite3_shim.c, which injects Chrome-specific
# definitions into the SQLite amalgamation code.
"amalgamation/sqlite3.c",
"src/amalgamation/sqlite3.c",
]
cflags = []
......@@ -355,8 +210,8 @@ component("chromium_sqlite3") {
}
include_dirs = [
".", # sqlite3.h here must override the one in amalgamation/.
"amalgamation",
".", # sqlite3.h here must override the one in src/amalgamation/.
"src/amalgamation",
]
configs -= [ "//build/config/compiler:chromium_code" ]
......@@ -390,14 +245,14 @@ component("dev_sqlite3") {
public = [ "dev/sqlite3.h" ]
sources = [
"amalgamation_dev/sqlite3.h",
"dev/sqlite3_shim.c",
"src/amalgamation_dev/sqlite3.h",
]
inputs = [
# This file is #included into sqlite3_shim.c, which injects Chrome-specific
# definitions into the SQLite amalgamation code.
"amalgamation_dev/sqlite3.c",
"src/amalgamation_dev/sqlite3.c",
]
cflags = []
......@@ -433,7 +288,7 @@ component("dev_sqlite3") {
include_dirs = [
"dev", # sqlite3.h here must override the one in amalgamation_dev/.
"amalgamation_dev",
"src/amalgamation_dev",
]
configs -= [ "//build/config/compiler:chromium_code" ]
......
third_party/sqlite/README.chromium
View file @ 591102e7
This diff is collapsed.
third_party/sqlite/README.md 0 → 100644
View file @ 591102e7
This is the top folder for Chromium's SQLite. The actual SQLite source is not
in this repository, but instead cloned into the `src` directory from
https://chromium.googlesource.com/chromium/deps/sqlite.
The directory structure is as follows. Files common to all third_party projects
(BUILD.GN, OWNERS, LICENSE) are omitted.
* `src/` The Chromium fork of SQLite (cloned via top level DEPS file).
* `scripts/` Scripts that generate the files in the amalgamations in src/.
* `sqlite.h` The header used by the rest of Chromium to include SQLite. This
forwards to src/amalgamation/sqlite3.h
* `fuzz/` Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite
build This directory contains:
The SQLite amalgamation is committed to the SQLite Chromium repository (in
`src`), but it is created by a script that lives in the Chromium repository.
This is because the configuration variables for building and amalgamation
generation are shared.
There are two amalgamations. The one in //third_party/sqlite/src/amalgamation
is used by Chromium. A second, located at
//third_party/sqlite/src/amalgamation_dev is used for some Chromium developer
tools and is not distributed.
# Upgrade to a new SQLite release.
**Note** SQLite tags all releases `version-<release number>`, e.g.
`version-3.31.1`. The Chromium project prefixes all tags/branches with
"chromium-", e.g. `chromium-version-3.31.1`.
1. Create new release branch
Use the SQLite commit ID when creating a branch. For example
"562fd18b9dc27216191c0a6477bba9b175f7f0d2" corresponds to the
3.31.1 release. The commit is used, instead of the tag name, because
we do not mirror the SQLite tags along with the commits. The correct
commit ID can be found at
[sqlite/releases](https://github.com/sqlite/sqlite/releases).
Create the branch at
[Gerrit/branches](https://chromium-review.googlesource.com/admin/repos/chromium/deps/sqlite,branches).
2. Checkout the new Chromium release branch.
```sh
cd //third_party/sqlite/src
git fetch origin
export VERSION=3.31.1
git checkout -b chromium-version-$VERSION \
--track origin/chromum-version-$VERSION
```
3. Generate and commit the SQLite amalgamations.
```sh
../scripts/generate_amalgamation.py
git add amalgamation
git add amalgamation_dev
git commit -m "Amalgamations for release $VERSION"
```
4. Run local tests.
Follow steps in [Running Tests](#running-tests) below to execute all
verifications and tests.
5. Upload the new release branch for review.
```sh
git cl upload
```
6. Roll the Chromium DEPS file.
Once review above has merged:
1. Roll the `chromium/src/DEPS` file to reference that new commit ID.
```sh
roll-dep src/third_party/sqlite/src
```
2. Update the version in //third_party/sqlite/README.chromium. Append the
commit created by roll-dep above.
# Cherry-pick unreleased commit from SQLite.
Sometimes **critical fixes** land in SQLite's master, but are not yet in a
release. If these need to be brought into the current release branch do the
following:
1. Checkout the current release branch.
```sh
export VERSION=3.31.1
cd //third_party/sqlite/src
git checkout -b chromium-version-$VERSION \
--track origin/chromium-version-$VERSION
```
2. Cherry-pick the change
Git _can_ be used to cherry pick upstream changes into a release branch but
the sqlite_cherry_picker.py script is preferred. This script automates a
few tasks such as:
* Identifying the correct Git commit ID to use if given the
Fossil commit ID.
* Automatically calculating Fossil manifest hashes.
* Skipping conflicted binary files.
* Generating the amalgamations.
Cherry-pick the commit:
```sh
../scripts/sqlite_cherry_picker.py <full git or fossil commit id>
```
If there is a conflict that the script cannot resolve then, like
git-cherry-pick, the script will exit and leave you to resolve the
conflicts. Once resolved run the script a second time:
```sh
../scripts/sqlite_cherry_picker.py --continue
```
3. Run local tests.
Follow steps in [Running Tests](#running-tests) below to execute all
verifications and tests.
4. Upload cherry-picked change (with amalgamations) for review.
```sh
git cl upload
```
5. Update the Chromium DEPS file.
Once review above has merged, roll the `chromium/src/DEPS` file to
reference that new commit ID.
```sh
roll-dep src/third_party/sqlite/src
```
# Running Tests
Build all desktop targets:
Check that `extract_sqlite_api.py` added "chrome_" to all exported symbols.
Only "_fini" and "_init" should be unprefixed, but are conditionally
exported by the linker and may be omitted.
```sh
autoninja -C out/Default
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
```
## Running unit tests
```sh
out/Default/sql_unittests
```
## Running web tests
```sh
third_party/blink/tools/run_web_tests.py -t Default storage/websql/
```
## Running SQLite's TCL test suite within the Chromium checkout.
This is one of the [SQLite test suites](https://www.sqlite.org/testing.html).
They take approximately 3 minutes to build and run on a fast workstation.
```sh
cd //third_party/sqlite
./scripts/generate_amalgamation.py --testing
make --directory=src test | tee /tmp/test.log
```
**Note**: Tests may fail on Chromium release branches. This is because some
tests rely on SQLite databases (binary files) which are committed to the
source and are likely not merged down when cherry picked. It is safe to
ignore these errors which should be reasonably easy to identify based on the
cherry picked upstream changes.
Show error'ed tests:
```sh
egrep 'errors out of' /tmp/test.log
```
Show broken tests:
```sh
egrep 'Failures on these tests:' /tmp/test.log
```
Broken tests will also show lines ending in "..." instead of "... Ok".
When done cleanup the SQLite repository:
```sh
cd src
make clean
git clean -i # and delete everything
rm -rf testdir
git checkout amalgamation/sqlite3.c
git checkout amalgamation_dev/sqlite3.c
```
third_party/sqlite/dev/sqlite3.h
View file @ 591102e7
......@@ -12,7 +12,7 @@
// other SQLite libraries loaded by the system libraries. This only matters when
// using the component build, where our SQLite's symbols are visible to the
// dynamic library loader.
#include "third_party/sqlite/amalgamation_dev/rename_exports.h"
#include "third_party/sqlite/amalgamation_dev/sqlite3.h"
#include "third_party/sqlite/src/amalgamation_dev/rename_exports.h"
#include "third_party/sqlite/src/amalgamation_dev/sqlite3.h"
#endif // THIRD_PARTY_SQLITE_DEV_SQLITE3_H_
third_party/sqlite/dev/sqlite3_shim.c
View file @ 591102e7
......@@ -2,15 +2,15 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This is a shim that injects Chrome-specific defitions into sqlite3.c
// This is a shim that injects Chrome-specific definitions into sqlite3.c
// BUILD.gn uses this instead of building the sqlite3 amalgamation directly.
// We prefix chrome_ to SQLite's exported symbols, so that we don't clash with
// other SQLite libraries loaded by the system libraries. This only matters when
// using the component build, where our SQLite's symbols are visible to the
// dynamic library loader.
#include "third_party/sqlite/amalgamation_dev/rename_exports.h"
#include "third_party/sqlite/src/amalgamation_dev/rename_exports.h"
#include "third_party/sqlite/sqlite3_shim_fixups.h"
#include "third_party/sqlite/amalgamation_dev/sqlite3.c"
#include "third_party/sqlite/src/amalgamation_dev/sqlite3.c"
third_party/sqlite/scripts/README_amalgamation.md 0 → 100644
View file @ 591102e7
**DO NOT EDIT FILES IN THIS DIRECTORY.**
These files are automatically generated from the SQLite originals. If files in
this directory are edited those changes **will** be dropped in subsequent
changes in this projects branch.
See the contents of `chromium/src/third_party/sqlite/scripts` for information
on how these files are built.
third_party/sqlite/scripts/generate_amalgamation.py 0 → 100755
View file @ 591102e7
#!/usr/bin/env python3
"""Creates the Chromium SQLite amalgamation.
The amalgamation is a single large source file (sqlite3.c) containing all
of the SQLite code. More at https://www.sqlite.org/amalgamation.html.
Usage:
generate_amalgamation.py
"""
import argparse
import os
import stat
import subprocess
import sys
import tempfile
from shutil import copyfile, rmtree
from extract_sqlite_api import ProcessSourceFile, header_line, footer_line
# The Chromium SQLite third party directory (i.e. //third_party/sqlite).
_SQLITE_ROOT_DIR = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
# The Chromium SQLite source directory (i.e. //third_party/sqlite/src).
_SQLITE_SRC_DIR = os.path.join(_SQLITE_ROOT_DIR, 'src')
# The .gni file (also used by BUILD.gn when building) which contains all
# flags passed to the `configuration` script and also used for the compile.
_COMMON_CONFIGURATION_FLAGS_GNI_FILE = os.path.join(
_SQLITE_ROOT_DIR, 'sqlite_common_configuration_flags.gni')
_CHROMIUM_CONFIGURATION_FLAGS_GNI_FILE = os.path.join(
_SQLITE_ROOT_DIR, 'sqlite_chromium_configuration_flags.gni')
_DEV_CONFIGURATION_FLAGS_GNI_FILE = os.path.join(
_SQLITE_ROOT_DIR, 'sqlite_dev_configuration_flags.gni')
# The temporary directory where `make configure` and the amalgamation
# is temporarily created.
_TEMP_CONFIG_DIR = tempfile.mkdtemp()
# Set to True to generate a configuration which is compatible for
# running the SQLite tests.
_CONFIGURE_FOR_TESTING = False
def get_amalgamation_dir(config_name):
if config_name == 'chromium':
return os.path.join(_SQLITE_SRC_DIR, 'amalgamation')
elif config_name == 'dev':
return os.path.join(_SQLITE_SRC_DIR, 'amalgamation_dev')
else:
assert False
def _icu_cpp_flags():
"""Return the libicu C++ flags."""
cmd = ['icu-config', '--cppflags']
try:
return subprocess.check_output(cmd)
except Exception:
return ''
def _icu_ld_flags():
"""Return the libicu linker flags."""
cmd = ['icu-config', '--ldflags']
try:
return subprocess.check_output(cmd)
except Exception:
return ''
def _strip_flags_for_testing(flags):
"""Accepts the default configure/build flags and strips out those
incompatible with the SQLite tests.
When configuring SQLite to run tests this script uses a configuration
as close to what Chromium ships as possible. Some flags need to be
omitted for the tests to link and run correct. See comments below.
"""
test_flags = []
for flag in flags:
# Omitting features can cause tests to hang/crash/fail because the
# SQLite tests don't seem to detect feature omission. Keep them enabled.
if flag.startswith('SQLITE_OMIT_'):
continue
# Some tests compile with specific SQLITE_DEFAULT_PAGE_SIZE so do
# not hard-code.
if flag.startswith('SQLITE_DEFAULT_PAGE_SIZE='):
continue
# Some tests compile with specific SQLITE_DEFAULT_MEMSTATUS so do
# not hard-code.
if flag.startswith('SQLITE_DEFAULT_MEMSTATUS='):
continue
# If enabled then get undefined reference to `uregex_open_63' and
# other *_64 functions.
if flag == 'SQLITE_ENABLE_ICU':
continue
# If defined then the fts4umlaut tests fail with the following error:
#
# Error: unknown tokenizer: unicode61
if flag == 'SQLITE_DISABLE_FTS3_UNICODE':
continue
test_flags.append(flag)
return test_flags
def _read_flags(file_name, param_name):
config_globals = dict()
with open(file_name) as input_file:
code = compile(input_file.read(), file_name, 'exec')
exec(code, config_globals)
return config_globals[param_name]
def _read_configuration_values(config_name):
"""Read the configuration flags and return them in an array.
|config_name| is one of "chromium" or "dev".
"""
common_flags = _read_flags(_COMMON_CONFIGURATION_FLAGS_GNI_FILE,
'sqlite_common_configuration_flags')
chromium_flags = _read_flags(_CHROMIUM_CONFIGURATION_FLAGS_GNI_FILE,
'sqlite_chromium_configuration_flags')
dev_flags = _read_flags(_DEV_CONFIGURATION_FLAGS_GNI_FILE,
'sqlite_dev_configuration_flags')
if config_name == 'chromium':
flags = common_flags + chromium_flags
elif config_name == 'dev':
flags = common_flags + dev_flags
else:
print('Incorrect config "%s"' % config_name, file=sys.stderr)
sys.exit(1)
if _CONFIGURE_FOR_TESTING:
flags = _strip_flags_for_testing(flags)
return flags
def _do_configure(config_name):
"""Run the configure script for the SQLite source."""
configure = os.path.join(_SQLITE_SRC_DIR, 'configure')
build_flags = ' '.join(
['-D' + f for f in _read_configuration_values(config_name)])
cflags = '-Os {} {}'.format(build_flags, _icu_cpp_flags())
ldflags = _icu_ld_flags()
cmd = [
configure,
'CFLAGS={}'.format(cflags),
'LDFLAGS={}'.format(ldflags),
'--disable-load-extension',
'--enable-amalgamation',
'--enable-threadsafe',
]
subprocess.check_call(cmd)
if _CONFIGURE_FOR_TESTING:
# Copy the files necessary for building/running tests back
#into the source directory.
files = ['Makefile', 'config.h', 'libtool']
for file_name in files:
copyfile(
os.path.join(_TEMP_CONFIG_DIR, file_name),
os.path.join(_SQLITE_SRC_DIR, file_name))
file_name = os.path.join(_SQLITE_SRC_DIR, 'libtool')
st = os.stat(file_name)
os.chmod(file_name, st.st_mode | stat.S_IEXEC)
def make_aggregate(config_name):
"""Generate the aggregate source files."""
if not os.path.exists(_TEMP_CONFIG_DIR):
os.mkdir(_TEMP_CONFIG_DIR)
try:
os.chdir(_TEMP_CONFIG_DIR)
_do_configure(config_name)
cmd = ['make', 'shell.c', 'sqlite3.h', 'sqlite3.c']
subprocess.check_call(cmd)
amalgamation_dir = get_amalgamation_dir(config_name)
if not os.path.exists(amalgamation_dir):
os.mkdir(amalgamation_dir)
readme_dst = os.path.join(amalgamation_dir, 'README.md')
if not os.path.exists(readme_dst):
readme_src = os.path.join(_SQLITE_ROOT_DIR, 'scripts',
'README_amalgamation.md')
copyfile(readme_src, readme_dst)
copyfile(
os.path.join(_TEMP_CONFIG_DIR, 'sqlite3.c'),
os.path.join(amalgamation_dir, 'sqlite3.c'))
copyfile(
os.path.join(_TEMP_CONFIG_DIR, 'sqlite3.h'),
os.path.join(amalgamation_dir, 'sqlite3.h'))
# shell.c must be placed in a different directory from sqlite3.h,
# because it contains an '#include "sqlite3.h"' that we want to resolve
# to our custom //third_party/sqlite/sqlite3.h, not to the sqlite3.h
# produced here.
shell_dir = os.path.join(amalgamation_dir, 'shell')
if not os.path.exists(shell_dir):
os.mkdir(shell_dir)
copyfile(
os.path.join(_TEMP_CONFIG_DIR, 'shell.c'),
os.path.join(shell_dir, 'shell.c'))
finally:
rmtree(_TEMP_CONFIG_DIR)
def extract_sqlite_api(config_name):
amalgamation_dir = get_amalgamation_dir(config_name)
input_file = os.path.join(amalgamation_dir, 'sqlite3.h')
output_file = os.path.join(amalgamation_dir, 'rename_exports.h')
ProcessSourceFile(
api_export_macro='SQLITE_API',
symbol_prefix='chrome_',
header_line=header_line,
footer_line=footer_line,
input_file=input_file,
output_file=output_file)
if __name__ == '__main__':
desc = \
('Create the SQLite amalgamation. The SQLite amalgamation is documented at '
'https://www.sqlite.org/amalgamation.html and is a single large file '
'containing the SQLite source code. Chromium generates the amalgamation with'
' this script to ensure that the configuration parameters are identical to '
'those in the Ninja build file.')
parser = argparse.ArgumentParser(description=desc)
parser.add_argument(
'-t',
'--testing',
action='store_true',
help='Generate an amalgamation for testing (default: false)')
namespace = parser.parse_args()
if namespace.testing:
_CONFIGURE_FOR_TESTING = True
print('Running configure for testing.')
for config_name in ['chromium', 'dev']:
make_aggregate(config_name)
extract_sqlite_api(config_name)
third_party/sqlite/scripts/generate_amalgamation.sh deleted 100755 → 0
View file @ e27a554c
#!/bin/sh
#
# Copyright (c) 2015 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# This script generates two amalgamated sources for SQLite.
# One is for Chromium, it has some features disabled. The other one is for
# developer tools, with a fuller set of features.
# The build flag sets should be in sync with the defines in BUILD.gn,
# "chromium_sqlite3_compile_options" and "common_sqlite3_compile_options"
# configs.
set -o errexit # Stop the script on the first error.
set -o nounset # Catch un-initialized variables.
cd patched
BUILD_GN_FLAGS_COMMON="\
-DSQLITE_DISABLE_FTS3_UNICODE \
-DSQLITE_DISABLE_FTS4_DEFERRED \
-DSQLITE_ENABLE_FTS3 \
-DSQLITE_ENABLE_ICU \
-DSQLITE_DEFAULT_FILE_PERMISSIONS=0600 \
-DSQLITE_DEFAULT_LOOKASIDE=0,0 \
-DSQLITE_DEFAULT_MEMSTATUS=1 \
-DSQLITE_DEFAULT_PAGE_SIZE=4096 \
-DSQLITE_DEFAULT_PCACHE_INITSZ=0 \
-DSQLITE_LIKE_DOESNT_MATCH_BLOBS \
-DSQLITE_HAVE_ISNAN \
-DSQLITE_MAX_WORKER_THREADS=0 \
-DSQLITE_MAX_MMAP_SIZE=268435456 \
-DSQLITE_OMIT_DECLTYPE \
-DSQLITE_OMIT_DEPRECATED \
-DSQLITE_OMIT_LOAD_EXTENSION \
-DSQLITE_OMIT_PROGRESS_CALLBACK \
-DSQLITE_OMIT_SHARED_CACHE \
-DSQLITE_SECURE_DELETE \
-DSQLITE_THREADSAFE=1 \
-DSQLITE_USE_ALLOCA \
"
BUILD_GN_FLAGS_CHROMIUM="\
-DSQLITE_OMIT_ANALYZE \
-DSQLITE_OMIT_AUTOINIT \
-DSQLITE_OMIT_AUTORESET \
-DSQLITE_OMIT_COMPILEOPTION_DIAGS \
-DSQLITE_OMIT_COMPLETE \
-DSQLITE_OMIT_EXPLAIN \
-DSQLITE_OMIT_GET_TABLE \
-DSQLITE_OMIT_LOOKASIDE \
-DSQLITE_OMIT_TCL_VARIABLE \
-DSQLITE_OMIT_REINDEX \
-DSQLITE_OMIT_TRACE \
-DSQLITE_OMIT_UPSERT \
-DSQLITE_OMIT_WINDOWFUNC \
"
BUILD_GN_FLAGS_DEV="\
-DSQLITE_ENABLE_EXPLAIN_COMMENTS \
"
generate_amalgamation() {
local build_flags=$1
local amalgamation_path=$2
mkdir build
cd build
../configure \
CFLAGS="-Os $build_flags $(icu-config --cppflags)" \
LDFLAGS="$(icu-config --ldflags)" \
--disable-load-extension \
--enable-amalgamation \
--enable-threadsafe
make shell.c sqlite3.h sqlite3.c
cp -f sqlite3.h sqlite3.c ../$amalgamation_path/
# shell.c must be placed in a different directory from sqlite3.h, because it
# contains an '#include "sqlite3.h"' that we want to resolve to our custom
# //third_party/sqlite/sqlite3.h, not to the sqlite3.h produced here.
mkdir -p ../$amalgamation_path/shell/
cp -f shell.c ../$amalgamation_path/shell/
cd ..
rm -rf build
../scripts/extract_sqlite_api.py $amalgamation_path/sqlite3.h \
$amalgamation_path/rename_exports.h
}
generate_amalgamation "$BUILD_GN_FLAGS_COMMON $BUILD_GN_FLAGS_CHROMIUM" ../amalgamation
generate_amalgamation "$BUILD_GN_FLAGS_COMMON $BUILD_GN_FLAGS_DEV" ../amalgamation_dev
third_party/sqlite/scripts/sqlite_cherry_picker.py 0 → 100755
View file @ 591102e7
This diff is collapsed.
third_party/sqlite/sqlite3.h
View file @ 591102e7
......@@ -12,7 +12,7 @@
// other SQLite libraries loaded by the system libraries. This only matters when
// using the component build, where our SQLite's symbols are visible to the
// dynamic library loader.
#include "third_party/sqlite/amalgamation/rename_exports.h"
#include "third_party/sqlite/src/amalgamation/rename_exports.h"
#if defined(SQLITE_OMIT_COMPLETE)
// When SQLITE_OMIT_COMPLETE is defined, sqlite3.h does not emit a declaration
......@@ -56,6 +56,6 @@
#endif // !defined(sqlite3_compileoption_used)
#endif // defined(SQLITE_OMIT_COMPILEOPTION_DIAGS)
#include "third_party/sqlite/amalgamation/sqlite3.h"
#include "third_party/sqlite/src/amalgamation/sqlite3.h"
#endif // THIRD_PARTY_SQLITE_SQLITE3_H_
third_party/sqlite/sqlite3_shim.c
View file @ 591102e7
......@@ -9,8 +9,8 @@
// other SQLite libraries loaded by the system libraries. This only matters when
// using the component build, where our SQLite's symbols are visible to the
// dynamic library loader.
#include "third_party/sqlite/amalgamation/rename_exports.h"
#include "third_party/sqlite/src/amalgamation/rename_exports.h"
#include "third_party/sqlite/sqlite3_shim_fixups.h"
#include "third_party/sqlite/amalgamation/sqlite3.c"
#include "third_party/sqlite/src/amalgamation/sqlite3.c"
third_party/sqlite/sqlite_chromium_configuration_flags.gni 0 → 100644
View file @ 591102e7
# Compile-time options passed to SQLite when compiling and building
# the Chromium amalgamation.
#
# The vast majority of the macros here are documented at
# https://www.sqlite.org/compile.html
#
# This file is read by a GN build script as well as generate_amalgamation.py
sqlite_chromium_configuration_flags = [
# Chrome doesn't use the ANALYZE SQLite extension.
#
# ANALYZE [1] is non-standard, and currently performs a table scan to
# update statistics used by the query planner. Chrome uses straightforward
# database schemas which do not require the level of fine tuning provided
# by ANALYZE, and we generally cannot afford the I/O cost of the required
# table scans.
#
# [1] https://www.sqlite.org/lang_analyze.html
"SQLITE_OMIT_ANALYZE",
# Chrome initializes SQLite manually in //sql/connection.cc.
"SQLITE_OMIT_AUTOINIT",
# Chrome calls sqlite3_reset() correctly to reset prepared statements.
"SQLITE_OMIT_AUTORESET",
# Chromium does not use sqlite3_{get,free}_table().
# Chrome doesn't use sqlite3_compileoption_{used,get}().
"SQLITE_OMIT_COMPILEOPTION_DIAGS",
# Chrome doesn't ship the SQLite shell, so command auto-completion is not
# needed. Chrome developers who build the SQLite shell living at
# //third_party/sqlite:sqlite_shell for diagnostic purposes will have to
# live without auto-completion.
"SQLITE_OMIT_COMPLETE",
# EXPLAIN's output is not stable across releases [1], so it should not be
# used in Chrome. Skipping the EXPLAIN machinery also results in
# non-trivial binary savings.
#
# [1] https://www.sqlite.org/lang_explain.html
"SQLITE_OMIT_EXPLAIN",
# Chrome does not use sqlite3_{get,free}_table().
"SQLITE_OMIT_GET_TABLE",
# Chrome already depends on malloc being very efficient, so we disable
# SQLite's arena allocator.
"SQLITE_DEFAULT_LOOKASIDE=0,0",
"SQLITE_OMIT_LOOKASIDE",
# Chrome doesn't use TCL variables.
"SQLITE_OMIT_TCL_VARIABLE",
# The REINDEX statemnt is only useful if a collation sequence's definition
# changes [1]. Chrome never defines its own collation sequences [2, 3], so
# it never needs to call REINDEX.
#
# [1] https://www.sqlite.org/lang_reindex.html
# [2] https://www.sqlite.org/datatype3.html#collating_sequences
# [3] https://www.sqlite.org/c3ref/create_collation.html
"SQLITE_OMIT_REINDEX",
# Chrome doesn't use sqlite3_{profile,trace}().
"SQLITE_OMIT_TRACE",
# Chrome doesn't use UPSERT.
"SQLITE_OMIT_UPSERT",
# Chrome doesn't use window functions in SQL.
"SQLITE_OMIT_WINDOWFUNC",
]
third_party/sqlite/sqlite_common_configuration_flags.gni 0 → 100644
View file @ 591102e7
# Compile-time options passed to SQLite when compiling and building
# the Chromium amalgamations.
sqlite_common_configuration_flags = [
"SQLITE_ENABLE_FTS3",
# New unicode61 tokenizer with built-in tables.
"SQLITE_DISABLE_FTS3_UNICODE",
# Chrome does not enable fts4, disable extra code.
"SQLITE_DISABLE_FTS4_DEFERRED",
"SQLITE_ENABLE_ICU",
# Defaults the secure_delete pragma to 1.
#
# This causes SQLite to overwrite all deleted information with zeroes,
# trading additional I/O for better privacy guarantees.
"SQLITE_SECURE_DELETE",
# TODO(pwnall): SQLite adds mutexes to protect structures which cross
# threads. In theory Chrome should be able to turn this to "2" which
# should give a slight speed boost. "2" is safe as long as a single
# connection is not used by more than one thread at a time.
"SQLITE_THREADSAFE=1",
# SQLite can spawn threads to sort in parallel if configured
# appropriately. Chrome doesn't configure SQLite for that, and would
# prefer to control distribution to worker threads.
"SQLITE_MAX_WORKER_THREADS=0",
# Allow 256MB mmap footprint per connection. Should not be too open-ended
# as that could cause memory fragmentation. 50MB encompasses the 99th
# percentile of Chrome databases in the wild.
# TODO(pwnall): A 64-bit-specific value could be 1G or more.
# TODO(pwnall): Figure out if exceeding this is costly.
"SQLITE_MAX_MMAP_SIZE=268435456",
# The default POSIX permissions for a newly created SQLite database.
#
# If unspecified, this defaults to 0644. All the data stored by Chrome is
# private, so our databases use stricter settings.
"SQLITE_DEFAULT_FILE_PERMISSIONS=0600",
# Needed by the SQL MemoryDumpProvider.
#
# Setting this to 1 is needed to collect the information reported by
# sqlite3_status64(SQLITE_STATUS_MEMORY_USED). Without this setting, the API
# still exists, but does not work as promised.
"SQLITE_DEFAULT_MEMSTATUS=1",
# Must match sql::Database::kDefaultPageSize.
"SQLITE_DEFAULT_PAGE_SIZE=4096",
# By default SQLite pre-allocates 100 pages of pcache data, which will not
# be released until the handle is closed. This is contrary to Chrome's
# memory-usage goals.
"SQLITE_DEFAULT_PCACHE_INITSZ=0",
# The flags below are recommended in the SQLite documentation.
"SQLITE_LIKE_DOESNT_MATCH_BLOBS",
"SQLITE_OMIT_DEPRECATED",
"SQLITE_OMIT_PROGRESS_CALLBACK",
"SQLITE_OMIT_SHARED_CACHE",
"SQLITE_USE_ALLOCA",
# Chrome does not use sqlite3_column_decltype().
"SQLITE_OMIT_DECLTYPE",
# Chrome does not use sqlite3_{enable_}load_extension().
# Asides from giving us fairly minor code savings, this option disables code
# that breaks our method for renaming SQLite's exported symbols. Last,
# there's a tiny security benefit to knowing that WebSQL can't possibly
# reach extension loading code.
"SQLITE_OMIT_LOAD_EXTENSION",
# Uses isnan() in the C99 standard library.
"SQLITE_HAVE_ISNAN",
]
third_party/sqlite/sqlite_dev_configuration_flags.gni 0 → 100644
View file @ 591102e7
# Compile-time options passed to SQLite when compiling and building
# the Chromium developer tools amalgamation.
#
# The vast majority of the macros here are documented at
# https://www.sqlite.org/compile.html
#
# This file is read by a GN build script as well as generate_amalgamation.py
sqlite_dev_configuration_flags = [ "SQLITE_ENABLE_EXPLAIN_COMMENTS" ]
third_party/sqlite/sqlite_shell_shim.c
View file @ 591102e7
......@@ -14,10 +14,10 @@
// While processing shell.c, rename main() to sqlite_shell_main().
#define main sqlite_shell_main
#include "third_party/sqlite/amalgamation/shell/shell.c"
#include "third_party/sqlite/src/amalgamation/shell/shell.c"
#undef main
int main(int argc, char** argv) {
InitializeICUForSqliteShell();
return sqlite_shell_main(argc, argv);
}
\ No newline at end of file
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment