Commit 3d8ec487 authored by Victor Costan's avatar Victor Costan Committed by Commit Bot

sqlite: Prefix SQLite API methods with chrome_.

In component builds, SQLite's API methods are exported from the
chromium_sqlite component, which means they are visible to the dynamic
library loader. This opens up the following possibilities:

1) A system library calls into our SQLite instead of calling into the
   system's SQLite library which it was built against. The patches in
   our SQLite version lead to different behavior from the system's
   SQLite, which can cause subtle failures. This happens if the dynamic
   library loader resolves the system library's symbol imports with our
   SQLite's exported symbols.
2) A system library loads the system SQLite, and we end up calling into
   it, instead of calling into our version of SQLite. This happens if
   the dynamic library loader resolves our symbol imports with the
   system's SQLite library.

Both possibilities above lead to the possibility that the component
build will behave differently from the release build, in subtle and
potentially non-deterministic ways. This is not a purely academic
concern. https://crbug.com/807487 happened because we use NSS on Linux,
and NSS invokes SQLite via a complex plugin system. On non-component builds,
NSS (a system library) loads and uses the system version of SQLite. On
component builds, NSS ends up using our SQLite.

This CL fixes the problem by adding a chrome_ prefix to all the symbols
exported from SQLite3. In C++ libraries, namespaces can make prefixing
easy. Unfortunately, SQLite is a C library, so the prefixing is fairly
heavy-handed. A high-level overview of the approach follows:

* An extract_sqlite_api Python script reads SQLite's header, extracts
  the names of all exported symbols, and writes a header file consisting
  of renaming preprocessor macros, e.g.
      #define sqlite3_init chrome_sqlite3_init
  David Benjamin <davidben@chromium.org> designed the approach and wrote
  the original version of the script.
* The script that we use to generate SQLite's amalgamation now also
  invokes the extract_sqlite_api script described above, and saves the
  output to amalgamation/rename_exports.h.
* The SQLite component exposes an sqlite3.h header that must be used by
  all SQLite3 users in Chromium. This header now #includes
  rename_exports.h (containing the renaming preprocessor macros) before
  #including amalgamation/sqlite3.h.
* sqlite3.c (the main output of the amalgamation process) does not
  #include "sqlite3.h". However, in order to facilitate autoconf builds,
  it does #include a "config.h", if a certain preprocessor define
  exists. We abuse that define to have sqlite.c always load config.h,
  and have config.h load our rename_exports.h.

This CL also adds a PRESUBMIT.py that runs unit tests for the
extract_sqlite_api Python script, which ensures that the script will not
break accidentally. Both the unit tests and the PRESUBIMT script are
inspired from //tools/vim.

Bug: 807093, 807487
Change-Id: If3868ba119ffd4ccbb06d1a6fcd4cc2ecd9ef2ae
Reviewed-on: https://chromium-review.googlesource.com/898549Reviewed-by: default avatarChris Mumford <cmumford@chromium.org>
Commit-Queue: Victor Costan <pwnall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534843}
parent 5157f928
......@@ -75,13 +75,14 @@ config("chromium_sqlite3_compile_options") {
# syntax working but execution failing. Review:
# src/src/parse.py
# src/tool/mkkeywordhash.c
]
# Pull in config.h on Linux. This allows use of preprocessor macros which
# are not available to the build config.
if (is_linux) {
defines += [ "_HAVE_SQLITE_CONFIG_H" ]
}
# Chromium does not use sqlite3_{enable_}load_extension().
# Asides from giving us fairly minor code savings, this option disables code
# that breaks our method for renaming SQLite's exported symbols. Last,
# there's a tiny security benefit to knowing that WebSQL can't possibly
# reach extension loading code.
"SQLITE_OMIT_LOAD_EXTENSION",
]
if (using_sanitizer) {
# Limit max length of data blobs and queries for fuzzing builds by 128 MB.
......@@ -138,7 +139,13 @@ if (!use_system_sqlite) {
]
cflags = []
defines = []
defines = [
# The generated sqlite3.c does not include sqlite3.h, so we cannot easily
# inject the renaming macros in amalgamation/export_renames.h. However,
# if the macro below is defined, sqlite3.c will #include "config.h", which
# can be used to inject the macros.
"_HAVE_SQLITE_CONFIG_H",
]
if (is_component_build) {
if (is_win) {
......@@ -168,7 +175,10 @@ if (!use_system_sqlite) {
]
}
include_dirs = [ "amalgamation" ]
include_dirs = [
".", # sqlite3.h here must override the one in amalgamation/.
"amalgamation",
]
configs -= [ "//build/config/compiler:chromium_code" ]
configs += [
......@@ -222,11 +232,14 @@ if (!use_system_sqlite) {
if (is_linux) {
executable("sqlite_shell") {
# So shell.c can find the correct sqlite3.h.
include_dirs = [ "amalgamation" ]
include_dirs = [
# shell.c contains an '#include "sqlite3.h", which we want to be
# resolved to //third_party/sqlite/shell.h.
".",
]
sources = [
"amalgamation/shell.c",
"amalgamation/shell/shell.c",
"src/src/shell_icu_linux.c",
# Include a dummy c++ file to force linking of libstdc++.
......
# Copyright 2018 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
"""Presubmit tests for /third_party/sqlite.
Runs Python unit tests in /third_party/sqlite/scripts on upload.
"""
def CheckChangeOnUpload(input_api, output_api):
results = []
results += input_api.RunTests(
input_api.canned_checks.GetUnitTests(input_api, output_api, [
'scripts/extract_sqlite_api_unittest.py'
]))
return results
......@@ -42,7 +42,7 @@ The directory structure is as follows. Files common to all third_party projects
build, which merges all the code in one .c file and one .h
file. See https://www.sqlite.org/amalgamation.html
* amalgamation/config.h - Linux build configuration
* google_generate_amalgamation.sh - Script that generates the amalgamation
* scripts/ - Scripts that generate the files in the amalgamation
* sqlite.h - The header used by the rest of Chromium to include SQLite. This
forwards to amalgamation/sqlite3.h
* fuzz/ - Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build
......@@ -54,7 +54,7 @@ The directory structure is as follows. Files common to all third_party projects
third_party/sqlite/src is the patched source from SQLite. This is used to
generate the amalgamation, a concatenation of all of the files into a giant
sqlite3.c. To prototype, edit in src/, then call
./google_generate_amalgamation.sh
./scripts/generate_amalgamation.sh
to regenerate sqlite3.c. The code in src/ is much easier to edit, and the
SQLite test framework can easily be run. During development it may be
convenient to modify BUILD.gn based on src/main.mk to just pull in the src/
......@@ -101,8 +101,8 @@ git add patches/*.patch
git commit -m "Rebuild patches for sqlite_${BASE}"
# Re-generate the amalgamation.
./google_generate_amalgamation.sh
git commit -m 'google_generate_amalgamation.sh' amalgamation/
./scripts/generate_amalgamation.sh
git commit -m './scripts/generate_amalgamation.sh' amalgamation/
# At this point everything should build and work.
# Do a squash upload. This should add your single patch to patches/, and apply
......@@ -204,16 +204,23 @@ git rebase sqlite-new-base
#### Finally, create the branch that we'll upload.
git new-branch --upstream-current sqlite-new-cl
./google_generate_amalgamation.sh
./scripts/generate_amalgamation.sh
#### Validate the upgrade.
# The goal is to have a set of reasonably-independent CLs which can be
# understood separately, so that future importers can sensibly determine how to
# handle conflicts. So use git-rebase and slipstream fixups back into their
# original CL until everything builds and works.
cd ../..
ninja -C out/Default
# Check that extract_sqlite_api.py added chrome_ to all exported symbols.
# Only "_fini" and "_init" should be unprefixed.
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
out/Default/sql_unittests
third_party/WebKit/Tools/Scripts/run-webkit-tests -t Default storage/websql/*
cd third_party/sqlite
#### Create the review.
# Rebuild the patch set.
git rm patches/*
git format-patch --output-directory=patches --ignore-space-change \
......
DO NOT EDIT FILES IN THIS DIRECTORY.
These files are automatically generated from the sqlite originals. If
you edit these files, your edits will be dropped in a future import of
the sqlite code.
See ../google_generate_amalgamation.sh for information on how these
files are built.
Scott Hess, April 6, 2011.
** DO NOT EDIT FILES IN THIS DIRECTORY. **
These files are automatically generated from the sqlite originals. If you
edit these files, your edits will be dropped in a future import of the sqlite
code.
See the contents of `../scripts` for information on how these files are
built.
\ No newline at end of file
/* On Windows and OSX, SQLite uses preprocessor macros to configure itself. On
* Linux, it expects config.h from autoconf. autoconf generates config.h by
* compiling a series of probe programs, and Chromium's build system has no
* "configure" phase to put such generation in. This file is a workaround for
* this issue.
*/
/* TODO(shess): Expand this to OSX and Windows? */
/* TODO(shess): Consider config_linux.h, config_mac.h, config_win.h? */
/* NOTE(shess): This file is included by sqlite3.c, be very careful about adding
* #include lines.
*/
/* TODO(shess): Consider using build/build_config.h for OS_ macros. */
/* TODO(shess): build_config.h uses unistd.h, perhaps for portability reasons,
* but AFAICT there are no current portability concerns here. limits.h is
* another alternative.
*/
// Copyright 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef THIRD_PARTY_SQLITE_AMALGAMATION_CONFIG_H_
#define THIRD_PARTY_SQLITE_AMALGAMATION_CONFIG_H_
// This file is included by sqlite3.c fairly early.
// We prefix chrome_ to SQLite's exported symbols, so that we don't clash with
// other SQLite libraries loaded by the system libraries. This only matters when
// using the component build, where our SQLite's symbols are visible to the
// dynamic library loader.
#include "third_party/sqlite/amalgamation/rename_exports.h"
// Linux-specific configuration fixups.
#if defined(__linux__)
// features.h, included below, indirectly includes sys/mman.h. The latter header
// only defines mremap if _GNU_SOURCE is defined. Depending on the order of the
// files in the amalgamation, removing the define below may result in a build
// error on Linux.
#if defined(__GNUC__) && !defined(_GNU_SOURCE)
# define _GNU_SOURCE
#define _GNU_SOURCE
#endif
#include <features.h>
/* SQLite wants to track malloc sizes. On OSX it uses malloc_size(), on
* Windows _msize(), elsewhere it handles it manually by enlarging the malloc
* and injecting a field. Enable malloc_usable_size() for Linux.
*
* malloc_usable_size() is not exported by the Android NDK. It is not
* implemented by uclibc.
*/
#if defined(__linux__) && !defined(__UCLIBC__)
// SQLite wants to track malloc sizes. On OSX it uses malloc_size(), on Windows
// _msize(), elsewhere it handles it manually by enlarging the malloc and
// injecting a field. Enable malloc_usable_size() for Linux.
//
// malloc_usable_size() is not exported by the Android NDK. It is not
// implemented by uclibc.
#if !defined(__UCLIBC__) && !defined(__ANDROID__)
#define HAVE_MALLOC_H 1
#define HAVE_MALLOC_USABLE_SIZE 1
#endif
/* TODO(shess): Eat other config options from gn and gyp? */
#endif // defined(__linux__)
#endif // THIRD_PARTY_SQLITE_AMALGAMATION_CONFIG_H_
// Copyright 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This file is generated by extract_sqlite_api.py.
#ifndef THIRD_PARTY_SQLITE_AMALGAMATION_RENAME_EXPORTS_H_
#define THIRD_PARTY_SQLITE_AMALGAMATION_RENAME_EXPORTS_H_
#define sqlite3_activate_cerod chrome_sqlite3_activate_cerod // Lines 5424-5426
#define sqlite3_activate_see chrome_sqlite3_activate_see // Lines 5414-5416
#define sqlite3_aggregate_context chrome_sqlite3_aggregate_context // Line 4961
#define sqlite3_aggregate_count chrome_sqlite3_aggregate_count // Line 4766
#define sqlite3_auto_extension chrome_sqlite3_auto_extension // Line 6055
#define sqlite3_backup_finish chrome_sqlite3_backup_finish // Line 7840
#define sqlite3_backup_init chrome_sqlite3_backup_init // Lines 7833-7838
#define sqlite3_backup_pagecount chrome_sqlite3_backup_pagecount // Line 7842
#define sqlite3_backup_remaining chrome_sqlite3_backup_remaining // Line 7841
#define sqlite3_backup_step chrome_sqlite3_backup_step // Line 7839
#define sqlite3_bind_blob chrome_sqlite3_bind_blob // Line 3979
#define sqlite3_bind_blob64 chrome_sqlite3_bind_blob64 // Lines 3980-3981
#define sqlite3_bind_double chrome_sqlite3_bind_double // Line 3982
#define sqlite3_bind_int chrome_sqlite3_bind_int // Line 3983
#define sqlite3_bind_int64 chrome_sqlite3_bind_int64 // Line 3984
#define sqlite3_bind_null chrome_sqlite3_bind_null // Line 3985
#define sqlite3_bind_parameter_count \
chrome_sqlite3_bind_parameter_count // Line 4014
#define sqlite3_bind_parameter_index \
chrome_sqlite3_bind_parameter_index // Line 4060
#define sqlite3_bind_parameter_name \
chrome_sqlite3_bind_parameter_name // Line 4042
#define sqlite3_bind_pointer chrome_sqlite3_bind_pointer // Line 3991
#define sqlite3_bind_text chrome_sqlite3_bind_text // Line 3986
#define sqlite3_bind_text16 chrome_sqlite3_bind_text16 // Line 3987
#define sqlite3_bind_text64 chrome_sqlite3_bind_text64 // Lines 3988-3989
#define sqlite3_bind_value chrome_sqlite3_bind_value // Line 3990
#define sqlite3_bind_zeroblob chrome_sqlite3_bind_zeroblob // Line 3992
#define sqlite3_bind_zeroblob64 chrome_sqlite3_bind_zeroblob64 // Line 3993
#define sqlite3_blob_bytes chrome_sqlite3_blob_bytes // Line 6596
#define sqlite3_blob_close chrome_sqlite3_blob_close // Line 6580
#define sqlite3_blob_open chrome_sqlite3_blob_open // Lines 6524-6532
#define sqlite3_blob_read chrome_sqlite3_blob_read // Line 6625
#define sqlite3_blob_reopen chrome_sqlite3_blob_reopen // Line 6557
#define sqlite3_blob_write chrome_sqlite3_blob_write // Line 6667
#define sqlite3_busy_handler chrome_sqlite3_busy_handler // Line 2386
#define sqlite3_busy_timeout chrome_sqlite3_busy_timeout // Line 2409
#define sqlite3_cancel_auto_extension \
chrome_sqlite3_cancel_auto_extension // Line 6067
#define sqlite3_changes chrome_sqlite3_changes // Line 2228
#define sqlite3_clear_bindings chrome_sqlite3_clear_bindings // Line 4070
#define sqlite3_close chrome_sqlite3_close // Line 331
#define sqlite3_close_v2 chrome_sqlite3_close_v2 // Line 332
#define sqlite3_collation_needed \
chrome_sqlite3_collation_needed // Lines 5363-5367
#define sqlite3_collation_needed16 \
chrome_sqlite3_collation_needed16 // Lines 5368-5372
#define sqlite3_column_blob chrome_sqlite3_column_blob // Line 4536
#define sqlite3_column_bytes chrome_sqlite3_column_bytes // Line 4543
#define sqlite3_column_bytes16 chrome_sqlite3_column_bytes16 // Line 4544
#define sqlite3_column_count chrome_sqlite3_column_count // Line 4086
#define sqlite3_column_database_name \
chrome_sqlite3_column_database_name // Line 4164
#define sqlite3_column_database_name16 \
chrome_sqlite3_column_database_name16 // Line 4165
#define sqlite3_column_decltype chrome_sqlite3_column_decltype // Line 4201
#define sqlite3_column_decltype16 chrome_sqlite3_column_decltype16 // Line 4202
#define sqlite3_column_double chrome_sqlite3_column_double // Line 4537
#define sqlite3_column_int chrome_sqlite3_column_int // Line 4538
#define sqlite3_column_int64 chrome_sqlite3_column_int64 // Line 4539
#define sqlite3_column_name chrome_sqlite3_column_name // Line 4115
#define sqlite3_column_name16 chrome_sqlite3_column_name16 // Line 4116
#define sqlite3_column_origin_name \
chrome_sqlite3_column_origin_name // Line 4168
#define sqlite3_column_origin_name16 \
chrome_sqlite3_column_origin_name16 // Line 4169
#define sqlite3_column_table_name chrome_sqlite3_column_table_name // Line 4166
#define sqlite3_column_table_name16 \
chrome_sqlite3_column_table_name16 // Line 4167
#define sqlite3_column_text chrome_sqlite3_column_text // Line 4540
#define sqlite3_column_text16 chrome_sqlite3_column_text16 // Line 4541
#define sqlite3_column_type chrome_sqlite3_column_type // Line 4545
#define sqlite3_column_value chrome_sqlite3_column_value // Line 4542
#define sqlite3_commit_hook chrome_sqlite3_commit_hook // Line 5670
#define sqlite3_compileoption_get chrome_sqlite3_compileoption_get // Line 191
#define sqlite3_compileoption_used \
chrome_sqlite3_compileoption_used // Line 190
#define sqlite3_complete chrome_sqlite3_complete // Line 2324
#define sqlite3_complete16 chrome_sqlite3_complete16 // Line 2325
#define sqlite3_config chrome_sqlite3_config // Line 1507
#define sqlite3_context_db_handle chrome_sqlite3_context_db_handle // Line 4988
#define sqlite3_create_collation \
chrome_sqlite3_create_collation // Lines 5313-5319
#define sqlite3_create_collation16 \
chrome_sqlite3_create_collation16 // Lines 5328-5334
#define sqlite3_create_collation_v2 \
chrome_sqlite3_create_collation_v2 // Lines 5320-5327
#define sqlite3_create_function \
chrome_sqlite3_create_function // Lines 4700-4709
#define sqlite3_create_function16 \
chrome_sqlite3_create_function16 // Lines 4710-4719
#define sqlite3_create_function_v2 \
chrome_sqlite3_create_function_v2 // Lines 4720-4730
#define sqlite3_create_module chrome_sqlite3_create_module // Lines 6327-6332
#define sqlite3_create_module_v2 \
chrome_sqlite3_create_module_v2 // Lines 6333-6339
#define sqlite3_data_count chrome_sqlite3_data_count // Line 4307
#define sqlite3_data_directory chrome_sqlite3_data_directory // Line 5541
#define sqlite3_db_cacheflush chrome_sqlite3_db_cacheflush // Line 8524
#define sqlite3_db_config chrome_sqlite3_db_config // Line 1526
#define sqlite3_db_filename chrome_sqlite3_db_filename // Line 5595
#define sqlite3_db_handle chrome_sqlite3_db_handle // Line 5578
#define sqlite3_db_mutex chrome_sqlite3_db_mutex // Line 6971
#define sqlite3_db_readonly chrome_sqlite3_db_readonly // Line 5605
#define sqlite3_db_release_memory chrome_sqlite3_db_release_memory // Line 5793
#define sqlite3_db_status chrome_sqlite3_db_status // Line 7200
#define sqlite3_declare_vtab chrome_sqlite3_declare_vtab // Line 6396
#define sqlite3_enable_load_extension \
chrome_sqlite3_enable_load_extension // Line 6017
#define sqlite3_enable_shared_cache \
chrome_sqlite3_enable_shared_cache // Line 5763
#define sqlite3_errcode chrome_sqlite3_errcode // Line 3403
#define sqlite3_errmsg chrome_sqlite3_errmsg // Line 3405
#define sqlite3_errmsg16 chrome_sqlite3_errmsg16 // Line 3406
#define sqlite3_errstr chrome_sqlite3_errstr // Line 3407
#define sqlite3_exec chrome_sqlite3_exec // Lines 403-409
#define sqlite3_expanded_sql chrome_sqlite3_expanded_sql // Line 3748
#define sqlite3_expired chrome_sqlite3_expired // Line 4767
#define sqlite3_extended_errcode chrome_sqlite3_extended_errcode // Line 3404
#define sqlite3_extended_result_codes \
chrome_sqlite3_extended_result_codes // Line 2103
#define sqlite3_file_control chrome_sqlite3_file_control // Line 7006
#define sqlite3_finalize chrome_sqlite3_finalize // Line 4573
#define sqlite3_free chrome_sqlite3_free // Line 2695
#define sqlite3_free_table chrome_sqlite3_free_table // Line 2492
#define sqlite3_get_autocommit chrome_sqlite3_get_autocommit // Line 5565
#define sqlite3_get_auxdata chrome_sqlite3_get_auxdata // Line 5047
#define sqlite3_get_table chrome_sqlite3_get_table // Lines 2484-2491
#define sqlite3_global_recover chrome_sqlite3_global_recover // Line 4769
#define sqlite3_initialize chrome_sqlite3_initialize // Line 1471
#define sqlite3_interrupt chrome_sqlite3_interrupt // Line 2289
#define sqlite3_key chrome_sqlite3_key // Lines 5382-5385
#define sqlite3_key_v2 chrome_sqlite3_key_v2 // Lines 5386-5390
#define sqlite3_last_insert_rowid chrome_sqlite3_last_insert_rowid // Line 2165
#define sqlite3_libversion chrome_sqlite3_libversion // Line 163
#define sqlite3_libversion_number chrome_sqlite3_libversion_number // Line 165
#define sqlite3_limit chrome_sqlite3_limit // Line 3475
#define sqlite3_load_extension chrome_sqlite3_load_extension // Lines 5985-5990
#define sqlite3_log chrome_sqlite3_log // Line 8061
#define sqlite3_malloc chrome_sqlite3_malloc // Line 2691
#define sqlite3_malloc64 chrome_sqlite3_malloc64 // Line 2692
#define sqlite3_memory_alarm chrome_sqlite3_memory_alarm // Lines 4771-4772
#define sqlite3_memory_highwater chrome_sqlite3_memory_highwater // Line 2722
#define sqlite3_memory_used chrome_sqlite3_memory_used // Line 2721
#define sqlite3_mprintf chrome_sqlite3_mprintf // Line 2598
#define sqlite3_msize chrome_sqlite3_msize // Line 2696
#define sqlite3_mutex_alloc chrome_sqlite3_mutex_alloc // Line 6816
#define sqlite3_mutex_enter chrome_sqlite3_mutex_enter // Line 6818
#define sqlite3_mutex_free chrome_sqlite3_mutex_free // Line 6817
#define sqlite3_mutex_held chrome_sqlite3_mutex_held // Line 6930
#define sqlite3_mutex_leave chrome_sqlite3_mutex_leave // Line 6820
#define sqlite3_mutex_notheld chrome_sqlite3_mutex_notheld // Line 6931
#define sqlite3_mutex_try chrome_sqlite3_mutex_try // Line 6819
#define sqlite3_next_stmt chrome_sqlite3_next_stmt // Line 5621
#define sqlite3_open chrome_sqlite3_open // Lines 3303-3306
#define sqlite3_open16 chrome_sqlite3_open16 // Lines 3307-3310
#define sqlite3_open_v2 chrome_sqlite3_open_v2 // Lines 3311-3316
#define sqlite3_os_end chrome_sqlite3_os_end // Line 1474
#define sqlite3_os_init chrome_sqlite3_os_init // Line 1473
#define sqlite3_overload_function chrome_sqlite3_overload_function // Line 6415
#define sqlite3_prepare chrome_sqlite3_prepare // Lines 3670-3676
#define sqlite3_prepare16 chrome_sqlite3_prepare16 // Lines 3692-3698
#define sqlite3_prepare16_v2 chrome_sqlite3_prepare16_v2 // Lines 3699-3705
#define sqlite3_prepare16_v3 chrome_sqlite3_prepare16_v3 // Lines 3706-3713
#define sqlite3_prepare_v2 chrome_sqlite3_prepare_v2 // Lines 3677-3683
#define sqlite3_prepare_v3 chrome_sqlite3_prepare_v3 // Lines 3684-3691
#define sqlite3_preupdate_count chrome_sqlite3_preupdate_count // Line 8623
#define sqlite3_preupdate_depth chrome_sqlite3_preupdate_depth // Line 8624
#define sqlite3_preupdate_hook chrome_sqlite3_preupdate_hook // Lines 8609-8621
#define sqlite3_preupdate_new chrome_sqlite3_preupdate_new // Line 8625
#define sqlite3_preupdate_old chrome_sqlite3_preupdate_old // Line 8622
#define sqlite3_profile chrome_sqlite3_profile // Lines 2946-2947
#define sqlite3_progress_handler chrome_sqlite3_progress_handler // Line 3074
#define sqlite3_randomness chrome_sqlite3_randomness // Line 2745
#define sqlite3_realloc chrome_sqlite3_realloc // Line 2693
#define sqlite3_realloc64 chrome_sqlite3_realloc64 // Line 2694
#define sqlite3_rekey chrome_sqlite3_rekey // Lines 5400-5403
#define sqlite3_rekey_v2 chrome_sqlite3_rekey_v2 // Lines 5404-5408
#define sqlite3_release_memory chrome_sqlite3_release_memory // Line 5779
#define sqlite3_reset chrome_sqlite3_reset // Line 4600
#define sqlite3_reset_auto_extension \
chrome_sqlite3_reset_auto_extension // Line 6075
#define sqlite3_result_blob chrome_sqlite3_result_blob // Line 5195
#define sqlite3_result_blob64 chrome_sqlite3_result_blob64 // Lines 5196-5197
#define sqlite3_result_double chrome_sqlite3_result_double // Line 5198
#define sqlite3_result_error chrome_sqlite3_result_error // Line 5199
#define sqlite3_result_error16 chrome_sqlite3_result_error16 // Line 5200
#define sqlite3_result_error_code chrome_sqlite3_result_error_code // Line 5203
#define sqlite3_result_error_nomem \
chrome_sqlite3_result_error_nomem // Line 5202
#define sqlite3_result_error_toobig \
chrome_sqlite3_result_error_toobig // Line 5201
#define sqlite3_result_int chrome_sqlite3_result_int // Line 5204
#define sqlite3_result_int64 chrome_sqlite3_result_int64 // Line 5205
#define sqlite3_result_null chrome_sqlite3_result_null // Line 5206
#define sqlite3_result_pointer chrome_sqlite3_result_pointer // Line 5214
#define sqlite3_result_subtype chrome_sqlite3_result_subtype // Line 5231
#define sqlite3_result_text chrome_sqlite3_result_text // Line 5207
#define sqlite3_result_text16 chrome_sqlite3_result_text16 // Line 5210
#define sqlite3_result_text16be chrome_sqlite3_result_text16be // Line 5212
#define sqlite3_result_text16le chrome_sqlite3_result_text16le // Line 5211
#define sqlite3_result_text64 chrome_sqlite3_result_text64 // Lines 5208-5209
#define sqlite3_result_value chrome_sqlite3_result_value // Line 5213
#define sqlite3_result_zeroblob chrome_sqlite3_result_zeroblob // Line 5215
#define sqlite3_result_zeroblob64 chrome_sqlite3_result_zeroblob64 // Line 5216
#define sqlite3_rollback_hook chrome_sqlite3_rollback_hook // Line 5671
#define sqlite3_rtree_geometry_callback \
chrome_sqlite3_rtree_geometry_callback // Lines 8872-8877
#define sqlite3_rtree_query_callback \
chrome_sqlite3_rtree_query_callback // Lines 8898-8904
#define sqlite3_set_authorizer chrome_sqlite3_set_authorizer // Lines 2836-2840
#define sqlite3_set_auxdata chrome_sqlite3_set_auxdata // Line 5048
#define sqlite3_set_last_insert_rowid \
chrome_sqlite3_set_last_insert_rowid // Line 2175
#define sqlite3_shutdown chrome_sqlite3_shutdown // Line 1472
#define sqlite3_sleep chrome_sqlite3_sleep // Line 5446
#define sqlite3_snapshot_cmp chrome_sqlite3_snapshot_cmp // Lines 8792-8795
#define sqlite3_snapshot_free chrome_sqlite3_snapshot_free // Line 8768
#define sqlite3_snapshot_get chrome_sqlite3_snapshot_get // Lines 8713-8717
#define sqlite3_snapshot_open chrome_sqlite3_snapshot_open // Lines 8751-8755
#define sqlite3_snapshot_recover chrome_sqlite3_snapshot_recover // Line 8817
#define sqlite3_snprintf chrome_sqlite3_snprintf // Line 2600
#define sqlite3_soft_heap_limit chrome_sqlite3_soft_heap_limit // Line 5857
#define sqlite3_soft_heap_limit64 chrome_sqlite3_soft_heap_limit64 // Line 5846
#define sqlite3_sourceid chrome_sqlite3_sourceid // Line 164
#define sqlite3_sql chrome_sqlite3_sql // Line 3747
#define sqlite3_status chrome_sqlite3_status // Line 7090
#define sqlite3_status64 chrome_sqlite3_status64 // Lines 7091-7096
#define sqlite3_step chrome_sqlite3_step // Line 4286
#define sqlite3_stmt_busy chrome_sqlite3_stmt_busy // Line 3805
#define sqlite3_stmt_readonly chrome_sqlite3_stmt_readonly // Line 3784
#define sqlite3_stmt_scanstatus \
chrome_sqlite3_stmt_scanstatus // Lines 8476-8481
#define sqlite3_stmt_scanstatus_reset \
chrome_sqlite3_stmt_scanstatus_reset // Line 8492
#define sqlite3_stmt_status chrome_sqlite3_stmt_status // Line 7343
#define sqlite3_strglob chrome_sqlite3_strglob // Line 7992
#define sqlite3_stricmp chrome_sqlite3_stricmp // Line 7974
#define sqlite3_strlike chrome_sqlite3_strlike // Line 8038
#define sqlite3_strnicmp chrome_sqlite3_strnicmp // Line 7975
#define sqlite3_system_errno chrome_sqlite3_system_errno // Line 8638
#define sqlite3_table_column_metadata \
chrome_sqlite3_table_column_metadata // Lines 5929-5939
#define sqlite3_temp_directory chrome_sqlite3_temp_directory // Line 5504
#define sqlite3_test_control chrome_sqlite3_test_control // Line 7025
#define sqlite3_thread_cleanup chrome_sqlite3_thread_cleanup // Line 4770
#define sqlite3_threadsafe chrome_sqlite3_threadsafe // Line 230
#define sqlite3_total_changes chrome_sqlite3_total_changes // Line 2252
#define sqlite3_trace chrome_sqlite3_trace // Lines 2944-2945
#define sqlite3_trace_v2 chrome_sqlite3_trace_v2 // Lines 3035-3040
#define sqlite3_transfer_bindings chrome_sqlite3_transfer_bindings // Line 4768
#define sqlite3_unlock_notify chrome_sqlite3_unlock_notify // Lines 7959-7963
#define sqlite3_update_hook chrome_sqlite3_update_hook // Lines 5722-5726
#define sqlite3_uri_boolean chrome_sqlite3_uri_boolean // Line 3358
#define sqlite3_uri_int64 chrome_sqlite3_uri_int64 // Line 3359
#define sqlite3_uri_parameter chrome_sqlite3_uri_parameter // Line 3357
#define sqlite3_user_data chrome_sqlite3_user_data // Line 4976
#define sqlite3_value_blob chrome_sqlite3_value_blob // Line 4874
#define sqlite3_value_bytes chrome_sqlite3_value_bytes // Line 4883
#define sqlite3_value_bytes16 chrome_sqlite3_value_bytes16 // Line 4884
#define sqlite3_value_double chrome_sqlite3_value_double // Line 4875
#define sqlite3_value_dup chrome_sqlite3_value_dup // Line 4915
#define sqlite3_value_free chrome_sqlite3_value_free // Line 4916
#define sqlite3_value_int chrome_sqlite3_value_int // Line 4876
#define sqlite3_value_int64 chrome_sqlite3_value_int64 // Line 4877
#define sqlite3_value_nochange chrome_sqlite3_value_nochange // Line 4887
#define sqlite3_value_numeric_type \
chrome_sqlite3_value_numeric_type // Line 4886
#define sqlite3_value_pointer chrome_sqlite3_value_pointer // Line 4878
#define sqlite3_value_subtype chrome_sqlite3_value_subtype // Line 4899
#define sqlite3_value_text chrome_sqlite3_value_text // Line 4879
#define sqlite3_value_text16 chrome_sqlite3_value_text16 // Line 4880
#define sqlite3_value_text16be chrome_sqlite3_value_text16be // Line 4882
#define sqlite3_value_text16le chrome_sqlite3_value_text16le // Line 4881
#define sqlite3_value_type chrome_sqlite3_value_type // Line 4885
#define sqlite3_version chrome_sqlite3_version // Line 162
#define sqlite3_vfs_find chrome_sqlite3_vfs_find // Line 6698
#define sqlite3_vfs_register chrome_sqlite3_vfs_register // Line 6699
#define sqlite3_vfs_unregister chrome_sqlite3_vfs_unregister // Line 6700
#define sqlite3_vmprintf chrome_sqlite3_vmprintf // Line 2599
#define sqlite3_vsnprintf chrome_sqlite3_vsnprintf // Line 2601
#define sqlite3_vtab_collation chrome_sqlite3_vtab_collation // Line 8371
#define sqlite3_vtab_config chrome_sqlite3_vtab_config // Line 8284
#define sqlite3_vtab_nochange chrome_sqlite3_vtab_nochange // Line 8356
#define sqlite3_vtab_on_conflict chrome_sqlite3_vtab_on_conflict // Line 8337
#define sqlite3_wal_autocheckpoint \
chrome_sqlite3_wal_autocheckpoint // Line 8132
#define sqlite3_wal_checkpoint chrome_sqlite3_wal_checkpoint // Line 8154
#define sqlite3_wal_checkpoint_v2 \
chrome_sqlite3_wal_checkpoint_v2 // Lines 8248-8254
#define sqlite3_wal_hook chrome_sqlite3_wal_hook // Lines 8097-8101
#define sqlite3changegroup_add chrome_sqlite3changegroup_add // Line 9826
#define sqlite3changegroup_add_strm \
chrome_sqlite3changegroup_add_strm // Lines 10250-10253
#define sqlite3changegroup_delete chrome_sqlite3changegroup_delete // Line 9861
#define sqlite3changegroup_new chrome_sqlite3changegroup_new // Line 9749
#define sqlite3changegroup_output \
chrome_sqlite3changegroup_output // Lines 9852-9856
#define sqlite3changegroup_output_strm \
chrome_sqlite3changegroup_output_strm // Lines 10254-10257
#define sqlite3changeset_apply \
chrome_sqlite3changeset_apply // Lines 10005-10019
#define sqlite3changeset_apply_strm \
chrome_sqlite3changeset_apply_strm // Lines 10206-10220
#define sqlite3changeset_concat \
chrome_sqlite3changeset_concat // Lines 9699-9706
#define sqlite3changeset_concat_strm \
chrome_sqlite3changeset_concat_strm // Lines 10221-10228
#define sqlite3changeset_conflict \
chrome_sqlite3changeset_conflict // Lines 9591-9595
#define sqlite3changeset_finalize chrome_sqlite3changeset_finalize // Line 9640
#define sqlite3changeset_fk_conflicts \
chrome_sqlite3changeset_fk_conflicts // Lines 9607-9610
#define sqlite3changeset_invert \
chrome_sqlite3changeset_invert // Lines 9670-9673
#define sqlite3changeset_invert_strm \
chrome_sqlite3changeset_invert_strm // Lines 10229-10234
#define sqlite3changeset_new chrome_sqlite3changeset_new // Lines 9564-9568
#define sqlite3changeset_next chrome_sqlite3changeset_next // Line 9440
#define sqlite3changeset_old chrome_sqlite3changeset_old // Lines 9531-9535
#define sqlite3changeset_op chrome_sqlite3changeset_op // Lines 9468-9474
#define sqlite3changeset_pk chrome_sqlite3changeset_pk // Lines 9501-9505
#define sqlite3changeset_start chrome_sqlite3changeset_start // Lines 9411-9415
#define sqlite3changeset_start_strm \
chrome_sqlite3changeset_start_strm // Lines 10235-10239
#define sqlite3session_attach chrome_sqlite3session_attach // Lines 9130-9133
#define sqlite3session_changeset \
chrome_sqlite3session_changeset // Lines 9257-9261
#define sqlite3session_changeset_strm \
chrome_sqlite3session_changeset_strm // Lines 10240-10244
#define sqlite3session_create chrome_sqlite3session_create // Lines 9004-9008
#define sqlite3session_delete chrome_sqlite3session_delete // Line 9022
#define sqlite3session_diff chrome_sqlite3session_diff // Lines 9319-9324
#define sqlite3session_enable chrome_sqlite3session_enable // Line 9042
#define sqlite3session_indirect chrome_sqlite3session_indirect // Line 9071
#define sqlite3session_isempty chrome_sqlite3session_isempty // Line 9376
#define sqlite3session_patchset \
chrome_sqlite3session_patchset // Lines 9355-9359
#define sqlite3session_patchset_strm \
chrome_sqlite3session_patchset_strm // Lines 10245-10249
#define sqlite3session_table_filter \
chrome_sqlite3session_table_filter // Lines 9144-9151
#endif // THIRD_PARTY_SQLITE_AMALGAMATION_RENAME_EXPORTS_H_
#!/usr/bin/env python
#
# Copyright 2018 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
'''
Parses SQLite source code and produces renaming macros for its exported symbols.
Usage:
extract_sqlite_api.py sqlite.h rename_macros.h
For example, the following renaming macro is produced for sqlite3_initialize().
#define sqlite3_initialize chrome_sqlite3_initialize
'''
import re
import sys
def ExtractLineTuples(string):
'''Returns a list of lines, with start/end whitespace stripped.
Each line is a tuple of (line number, string).
'''
raw_lines = string.split('\n')
stripped_lines = [line.strip() for line in raw_lines]
return list(enumerate(stripped_lines, start=1))
def ExtractPreprocessorDirectives(lines):
'''Extracts preprocessor directives from lines of C code.
Each input line should be a tuple of (line number, string).
Returns a list of preprocessor directives, and a list of C code lines with the
preprocessor directives removed. The returned code lines are a subset of the
input tuples.
'''
code_lines = []
directives = []
in_directive = False
last_directive = []
for line_tuple in lines:
line = line_tuple[1]
# Preprocessor directives start with #.
if not in_directive:
if len(line) > 0 and line[0] == '#':
in_directive = True
last_directive = []
# Preprocessor directives use \ as a line continuation character.
if in_directive:
if line[-1] == '\\':
line = line[:-1]
else:
in_directive = False
last_directive.append(line)
if not in_directive:
directives.append('\n'.join(last_directive))
else:
code_lines.append(line_tuple)
return directives, code_lines
# Regular expression used to parse a macro definition.
DEFINITION_RE = re.compile(r'^\#\s*define\s+(\w+)(\s|$)')
def ExtractDefineMacroName(line):
'''Extracts the macro name from a non-function preprocessor definition.
Returns None if the preprocessor line is not a preprocessor macro definition.
Macro functions are not considered preprocessor definitions.
'''
match = DEFINITION_RE.match(line)
if match is None:
return None
return match.group(1)
# Matches C++-style // single-line comments.
SINGLE_LINE_COMMENT_RE = re.compile(r'//.*$')
# Matches C-style /* multi-line comments */.
MULTI_LINE_COMMENT_RE = re.compile(r'/\*.*?\*/', flags=re.MULTILINE|re.DOTALL)
def RemoveLineComments(line):
'''Returns the given C code line with comments removed.
This handles both C-style /* comments */ and C++-style // comments, but cannot
tackle C-style comments that extend over multiple lines.
'''
return SINGLE_LINE_COMMENT_RE.sub('', MULTI_LINE_COMMENT_RE.sub('', line))
def RemoveComments(code_tuples):
'Returns the given C code tuples with all comments removed.'
output_tuples = []
in_comment = False
for line_number, line in code_tuples:
if in_comment:
if '*/' in line:
_, line = line.split('*/', 1)
in_comment = False
if not in_comment:
line = RemoveLineComments(line)
if '/*' in line:
line, _ = line.split('/*', 1)
in_comment = True
output_tuples.append((line_number, line))
return output_tuples
# Splits a line of C code into statement pieces.
STATEMENT_BREAK_RE = re.compile(r'[;{}]')
def ToStatementTuples(code_tuples):
'''Converts C code lines into statements.
The input is tuples of (line number, line code string). The output is
tuples of (min line, max line, statement).
The function considers ; { and } to be statement separators. This is
sufficiently correct, given our goal.
'''
statements = []
current_statement = ''
current_start = 0
for line_number, line in code_tuples:
pieces = STATEMENT_BREAK_RE.split(line)
for piece in pieces[:-1]: # The last piece is an unfinished statement.
if current_statement != '':
current_statement = current_statement + '\n' + piece
statements.append(
(current_start, line_number, current_statement.strip()))
current_statement = ''
else:
statements.append((line_number, line_number, piece.strip()))
if current_statement == '':
current_start = line_number
if pieces[-1] != '':
current_statement = current_statement + '\n' + pieces[-1]
return statements
# Used to break down a line into words.
WHITESPACE_RE = re.compile(r'\s+')
# Features unsupported by our extractor.
#
# We do not support parsing struct and enum literals because sqlite typedefs
# them before incorporating them into exported symbols. We can avoid matching
# curly braces because we do not support enum, struct, or union, and we only
# need to consider declarations involving typedef names and primitive types.
UNSUPPORTED_KEYWORDS = set([
'enum',
'struct',
'union',
'typedef'
])
# Type qualifiers that we can skip over.
#
# We discard storage-class specifiers and type qualifiers. For purposes of
# finding the end of declaration specifiers, they are not needed. This
# additionally discards any pointer type qualifiers.
QUALIFIER_KEYWORDS = set([
'extern',
'static',
'auto',
'register',
'const',
'volatile',
])
# Keywords used in composite primitive types.
#
# Types using these keywords may have more than one keyword, e.g.
# "long long int".
COMPOSITE_TYPE_SPECIFIERS = set([
'char',
'short',
'int',
'long',
'float',
'double',
'signed',
'unsigned',
])
# Matches an identifier.
IDENTIFIER_RE = re.compile(r'^[a-zA-Z_0-9]+$')
def ExtractApiExport(macro_names, api_export_macro, statement):
'''Extracts the symbol name from a statement exporting a function.
Returns None if the statement does not export a symbol. Throws ValueError if
the parser cannot understand the statement.
'''
# See http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf, section 6.7
# for how to parse C declarations. Note that a declaration is a number of
# declaration-specifiers, followed by a list of declarators with optional
# initializer. Multiple declarators would be a declaration like:
#
# int a, b;
#
# While, in principle, one could declare a pair of C functions like this, no
# one does it. We assume there is only one declarator.
#
# int foo(int), bar(int, int);
#
# Jumping to section 6.7.5, a declarator includes some optional pointer
# specifiers (which may have type qualifiers like 'const' embedded, e.g. 'int
# * const * const foo') and some grouping. Note, however, that in all cases,
# the declaration name is the first non-type-qualifier identifier.
#
# Thus our goal is to skip the declaration specifiers and get to the
# declarators.
# Simplification: get rid of pointer characters.
statement = statement.replace('*', ' ')
# Simplification: make sure each open parenthesis is each own word.
statement = statement.replace('(', ' ( ')
statement = statement.replace('[', ' [ ')
words = WHITESPACE_RE.split(statement)
# Ignore statements that don't deal with exporting symbols.
if api_export_macro not in words:
return None
seen_composite_type = False
seen_simple_type = False
for word in words:
if word in UNSUPPORTED_KEYWORDS:
raise ValueError("Unsupported keyword %s" % word)
if word in QUALIFIER_KEYWORDS:
continue
# Per section 6.7.2, we must have at least one type specifier (so the first
# token is one). Moreover, clause 2 implies that if we have a typedef name,
# enum, struct, or union, it is the only type specifier. If we have a
# keyword such as 'int', we may have one or more of such keywords.
if word in COMPOSITE_TYPE_SPECIFIERS:
if seen_simple_type:
raise ValueError('Mixed simple (struct_name) and composite (int) types')
seen_composite_type = True
continue
# We assume that macros are only used for qualifiers, which can be skipped.
if word in macro_names or word == api_export_macro:
continue
if not seen_composite_type and not seen_simple_type:
seen_simple_type = True
if IDENTIFIER_RE.match(word) is None:
raise ValueError(
"%s parsed as type name, which doesn't make sense" % word)
continue
if IDENTIFIER_RE.match(word) is None:
raise ValueError(
"%s parsed as symbol name, which doesn't make sense" % word)
return word
raise ValueError('Failed to find symbol name')
def ExportedSymbolLine(symbol_prefix, symbol, statement_tuple):
'Returns an output line for an exported symbol.'
if statement_tuple[0] == statement_tuple[1]:
lines = 'Line %d' % statement_tuple[0]
else:
lines = 'Lines %d-%d' % (statement_tuple[0], statement_tuple[1])
return '#define %s %s%s // %s' % (symbol, symbol_prefix, symbol, lines)
def ExportedExceptionLine(exception, statement_tuple):
'Returns an output line for a parsing failure.'
# Output a TODO without a name so the broken parsing result doesn't
# accidentally get checked in.
return '// TODO: Lines %d-%d -- %s' % (
statement_tuple[0], statement_tuple[1], exception.message)
def ProcessSource(api_export_macro, symbol_prefix, header_line, footer_line,
file_content):
'Returns a list of lines that rename exported symbols in an C program file.'
line_tuples = ExtractLineTuples(file_content)
line_tuples = RemoveComments(line_tuples)
directives, code_tuples = ExtractPreprocessorDirectives(line_tuples)
macro_names = set(
name for name in
[ExtractDefineMacroName(directive) for directive in directives]
if name is not None)
statement_tuples = ToStatementTuples(code_tuples)
output_lines = []
for statement_tuple in statement_tuples:
line = statement_tuple[2]
try:
symbol_name = ExtractApiExport(macro_names, api_export_macro, line)
if symbol_name:
output_lines.append(
ExportedSymbolLine(symbol_prefix, symbol_name, statement_tuple))
except ValueError as exception:
output_lines.append(ExportedExceptionLine(exception, statement_tuple))
output_lines.sort()
return [header_line] + output_lines + [footer_line]
def ProcessSourceFile(api_export_macro, symbol_prefix, header_line,
footer_line, input_file, output_file):
'Reads in a C program file and outputs macros renaming exported symbols.'
with open(input_file, 'r') as f:
file_content = f.read()
output_lines = ProcessSource(api_export_macro, symbol_prefix, header_line,
footer_line, file_content)
output_lines.append('')
with open(output_file, 'w') as f:
f.write('\n'.join(output_lines))
header_line='''// Copyright 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This file is generated by extract_sqlite_api.py.
#ifndef THIRD_PARTY_SQLITE_AMALGAMATION_RENAME_EXPORTS_H_
#define THIRD_PARTY_SQLITE_AMALGAMATION_RENAME_EXPORTS_H_
'''
footer_line ='''
#endif // THIRD_PARTY_SQLITE_AMALGAMATION_RENAME_EXPORTS_H_
'''
if __name__ == '__main__':
ProcessSourceFile(api_export_macro='SQLITE_API', symbol_prefix='chrome_',
header_line=header_line, footer_line=footer_line,
input_file=sys.argv[1], output_file=sys.argv[2])
#!/usr/bin/env python
# Copyright 2018 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
"""Tests for extract_sqlite_api.py.
These tests should be getting picked up by the PRESUBMIT.py in this directory.
"""
import imp
import os
import shutil
import sys
import tempfile
import unittest
class ExtractSqliteApiUnittest(unittest.TestCase):
def setUp(self):
self.test_root = tempfile.mkdtemp()
source_path = os.path.join(
os.path.dirname(os.path.realpath(__file__)), 'extract_sqlite_api.py')
self.extractor = imp.load_source('extract_api', source_path)
def tearDown(self):
if self.test_root:
shutil.rmtree(self.test_root)
def testExtractLineTuples(self):
golden = [(1, 'Line1'), (2, ''), (3, 'Line 2'), (4, 'Line3'), (5, '')]
text_with_newline = "Line1\n\nLine 2 \nLine3\n"
self.assertEqual(self.extractor.ExtractLineTuples(text_with_newline),
golden)
golden = [(1, 'Line1'), (2, ''), (3, 'Line 2'), (4, 'Line3')]
text_without_newline = "Line1\n\nLine 2 \nLine3"
self.assertEqual(self.extractor.ExtractLineTuples(text_without_newline),
golden)
def testExtractPreprocessorDirectives(self):
lines = [
(1, '// Header comment'),
(2, '#define DIRECTIVE 1'),
(3, 'int main() { // \\'),
(4, '}'),
(5, ''),
(6, '#define MULTILINE \\'),
(7, 'MORE_MULTILINE_DIRECTIVE\\'),
(8, 'END_MULTILINE_DIRECTIVE'),
(9, 'void code() { }'),
]
directives, code_lines = self.extractor.ExtractPreprocessorDirectives(lines)
self.assertEqual(directives, [
'#define DIRECTIVE 1',
'#define MULTILINE \nMORE_MULTILINE_DIRECTIVE\nEND_MULTILINE_DIRECTIVE',
])
self.assertEqual(code_lines, [
(1, '// Header comment'),
(3, 'int main() { // \\'),
(4, '}'),
(5, ''),
(9, 'void code() { }'),
])
def testExtractDefineMacroName(self):
self.assertEqual(
'SQLITE_API', self.extractor.ExtractDefineMacroName(
'#define SQLITE_API 1'))
self.assertEqual(
'SQLITE_API', self.extractor.ExtractDefineMacroName(
'#define SQLITE_API'))
self.assertEqual(
'SQLITE_API', self.extractor.ExtractDefineMacroName(
'#define SQLITE_API\n1'))
self.assertEqual(
'SQLITE_API', self.extractor.ExtractDefineMacroName(
'# define SQLITE_API 1'))
self.assertEqual(
'SQLITE_API', self.extractor.ExtractDefineMacroName(
'#\tdefine\tSQLITE_API\t1'))
self.assertEqual(
None, self.extractor.ExtractDefineMacroName(
' #define SQLITE_API 1'))
self.assertEqual(
None, self.extractor.ExtractDefineMacroName(
' #define SQLITE_API() 1'))
self.assertEqual(None, self.extractor.ExtractDefineMacroName(''))
def testRemoveLineComments(self):
self.assertEqual(
'word;', self.extractor.RemoveLineComments('word;'))
self.assertEqual(
'', self.extractor.RemoveLineComments(''))
self.assertEqual(
'', self.extractor.RemoveLineComments('// comment'))
self.assertEqual(
'', self.extractor.RemoveLineComments('/* comment */'))
self.assertEqual(
'word;', self.extractor.RemoveLineComments('wo/*comment*/rd;'))
self.assertEqual(
'word;*/', self.extractor.RemoveLineComments('wo/*comment*/rd;*/'))
self.assertEqual(
'word;*/', self.extractor.RemoveLineComments('wo/*/*comment*/rd;*/'))
self.assertEqual(
'word;', self.extractor.RemoveLineComments('wo/*comm//ent*/rd;'))
def testRemoveComments(self):
lines = [
(1, 'code();'),
(2, 'more_code(); /* with comment */ more_code();'),
(3, '/**'),
(4, 'Spec text'),
(5, '**/ spec_code();'),
(6, 'late_code(); /* with comment */ more_late_code(); /* late comment'),
(7, 'ends here // C++ trap */ code(); // /* C trap'),
(8, 'last_code();'),
]
self.assertEqual(self.extractor.RemoveComments(lines), [
(1, 'code();'),
(2, 'more_code(); more_code();'),
(3, ''),
(5, ' spec_code();'),
(6, 'late_code(); more_late_code(); '),
(7, ' code(); '),
(8, 'last_code();'),
])
def testToStatementTuples(self):
lines = [
(1, 'void function();'),
(2, 'int main('),
(3, ' int argc, char* argv) {'),
(4, ' statement1; statement2;'),
(5, '}'),
(6, 'stat'),
(7, 'ement4; statement5; sta'),
(8, 'tem'),
(9, 'ent6; statement7;')
]
self.assertEqual(self.extractor.ToStatementTuples(lines), [
(1, 1, 'void function()'),
(2, 3, 'int main(\n int argc, char* argv)'),
(4, 4, 'statement1'),
(4, 4, 'statement2'),
(5, 5, ''),
(6, 7, 'stat\nement4'),
(7, 7, 'statement5'),
(7, 9, 'sta\ntem\nent6'),
(9, 9, 'statement7'),
])
def testExtractApiExport(self):
self.assertEqual(
'sqlite3_init',
self.extractor.ExtractApiExport(
set(), 'SQLITE_API', 'SQLITE_API void sqlite3_init()'))
self.assertEqual(
'sqlite3_sleep',
self.extractor.ExtractApiExport(
set(), 'SQLITE_API', 'SQLITE_API int sqlite3_sleep(int ms)'))
self.assertEqual(
'sqlite3_sleep',
self.extractor.ExtractApiExport(
set(), 'SQLITE_API',
'SQLITE_API long long sqlite3_sleep(int ms)'))
self.assertEqual(
'sqlite3rbu_temp_size',
self.extractor.ExtractApiExport(
set(), 'SQLITE_API',
'SQLITE_API sqlite3_int64 sqlite3rbu_temp_size(sqlite3rbu *pRbu)'))
self.assertEqual(
'sqlite3_expired',
self.extractor.ExtractApiExport(
set(['SQLITE_DEPRECATED']), 'SQLITE_API',
'SQLITE_API SQLITE_DEPRECATED int sqlite3_expired(sqlite3_stmt*)'))
# SQLite's header actually #defines double (in some cases).
self.assertEqual(
'sqlite3_column_double',
self.extractor.ExtractApiExport(
set(['double']), 'SQLITE_API',
'SQLITE_API double sqlite3_column_double(sqlite3_stmt*, int iCol)'))
self.assertEqual(
'sqlite3_temp_directory',
self.extractor.ExtractApiExport(
set(['SQLITE_EXTERN']), 'SQLITE_API',
'SQLITE_API SQLITE_EXTERN char *sqlite3_temp_directory'))
self.assertEqual(
'sqlite3_version',
self.extractor.ExtractApiExport(
set(['SQLITE_EXTERN']), 'SQLITE_API',
'SQLITE_API SQLITE_EXTERN const char sqlite3_version[]'))
self.assertEqual(
None,
self.extractor.ExtractApiExport(
set(['SQLITE_DEPRECATED']), 'SQLITE_API',
'NOT_SQLITE_API struct sqlite_type sqlite3_sleep(int ms)'))
with self.assertRaisesRegexp(ValueError, 'Mixed simple .* and composite'):
self.extractor.ExtractApiExport(
set(), 'SQLITE_API', 'SQLITE_API void int sqlite3_sleep(int ms)')
with self.assertRaisesRegexp(ValueError, 'Unsupported keyword struct'):
self.extractor.ExtractApiExport(
set(), 'SQLITE_API',
'SQLITE_API struct sqlite_type sqlite3_sleep(int ms)')
with self.assertRaisesRegexp(ValueError, 'int\+\+ parsed as type name'):
self.extractor.ExtractApiExport(
set(), 'SQLITE_API', 'SQLITE_API int++ sqlite3_sleep(int ms)')
with self.assertRaisesRegexp(ValueError, 'sqlite3\+sleep parsed as symbol'):
self.extractor.ExtractApiExport(
set(), 'SQLITE_API', 'SQLITE_API int sqlite3+sleep(int ms)')
def testExportedSymbolLine(self):
self.assertEqual(
'#define sqlite3_sleep chrome_sqlite3_sleep // Line 42',
self.extractor.ExportedSymbolLine(
'chrome_', 'sqlite3_sleep',
(42, 42, 'SQLITE_API int chrome_sqlite3_sleep(int ms)')))
self.assertEqual(
'#define sqlite3_sleep chrome_sqlite3_sleep // Lines 42-44',
self.extractor.ExportedSymbolLine(
'chrome_', 'sqlite3_sleep',
(42, 44, 'SQLITE_API int chrome_sqlite3_sleep(int ms)')))
def testExportedExceptionLine(self):
self.assertEqual(
'// TODO: Lines 42-44 -- Something went wrong',
self.extractor.ExportedExceptionLine(
ValueError('Something went wrong'),
(42, 44, 'SQLITE_API int chrome_sqlite3_sleep(int ms)')))
def testProcessSource(self):
file_content = '\n'.join([
'/*',
'struct sqlite_type sqlite3_sleep; // Remove comments',
'*/',
'#define SQLITE_DEPRECATED',
'SQLITE_API int sqlite3_sleep(int ms);',
'SQLITE_API struct sqlite_type sqlite3_sleep(int ms);',
'SQLITE_API SQLITE_DEPRECATED int sqlite3_expired(sqlite3_stmt*);',
])
golden_output = [
'// Header',
'#define sqlite3_expired chrome_sqlite3_expired // Line 7',
'#define sqlite3_sleep chrome_sqlite3_sleep // Line 5',
'// TODO: Lines 6-6 -- Unsupported keyword struct',
'// Footer',
]
self.assertEqual(
golden_output,
self.extractor.ProcessSource('SQLITE_API', 'chrome_', '// Header',
'// Footer', file_content))
def testProcessSourceFile(self):
file_content = '\n'.join([
'/*',
'struct sqlite_type sqlite3_sleep; // Remove comments',
'*/',
'#define SQLITE_DEPRECATED',
'SQLITE_API int sqlite3_sleep(int ms);',
'SQLITE_API struct sqlite_type sqlite3_sleep(int ms);',
'SQLITE_API SQLITE_DEPRECATED int sqlite3_expired(sqlite3_stmt*);',
])
golden_output = '\n'.join([
'// Header',
'#define sqlite3_expired chrome_sqlite3_expired // Line 7',
'#define sqlite3_sleep chrome_sqlite3_sleep // Line 5',
'// TODO: Lines 6-6 -- Unsupported keyword struct',
'// Footer',
'',
])
input_file = os.path.join(self.test_root, 'input.h')
output_file = os.path.join(self.test_root, 'macros.h')
with open(input_file, 'w') as f:
f.write(file_content)
self.extractor.ProcessSourceFile(
'SQLITE_API', 'chrome_', '// Header', '// Footer', input_file,
output_file)
with open(output_file, 'r') as f:
self.assertEqual(f.read(), golden_output)
if __name__ == '__main__':
unittest.main()
......@@ -9,10 +9,19 @@ cd src
mkdir bld
cd bld
../configure
FILES="shell.c sqlite3.h sqlite3.c"
OPTS=""
make "OPTS=$OPTS" $FILES
cp -f $FILES ../../amalgamation
make "OPTS=$OPTS" shell.c sqlite3.h sqlite3.c
cp -f sqlite3.h sqlite3.c ../../amalgamation
# shell.c must be placed in a different directory from sqlite3.h, because it
# contains an '#include "sqlite3.h"' that we want to resolve to our custom
# //third_party/sqlite/sqlite3.h, not to the sqlite3.h produced here.
mkdir -p ../../amalgamation/shell/
cp -f shell.c ../../amalgamation/shell/
cd ..
rm -rf bld
../scripts/extract_sqlite_api.py ../amalgamation/sqlite3.h \
../amalgamation/rename_exports.h
\ No newline at end of file
......@@ -4,15 +4,21 @@
#ifndef THIRD_PARTY_SQLITE_SQLITE3_H_
#define THIRD_PARTY_SQLITE_SQLITE3_H_
#pragma once
// This is a shim header to include the right sqlite3 header.
// Use this instead of referencing the sqlite3 header directly.
// This is a shim header to include the right sqlite3 headers.
// Use this instead of referencing sqlite3 headers directly.
#if defined(USE_SYSTEM_SQLITE)
#include <sqlite3.h>
#else
// We prefix chrome_ to SQLite's exported symbols, so that we don't clash with
// other SQLite libraries loaded by the system libraries. This only matters when
// using the component build, where our SQLite's symbols are visible to the
// dynamic library loader.
#include "third_party/sqlite/amalgamation/rename_exports.h"
#include "third_party/sqlite/amalgamation/sqlite3.h"
#endif
#endif // defined(USE_SYSTEM_SQLITE)
#endif // THIRD_PARTY_SQLITE_SQLITE3_H_
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment