Skip to content

GitLab

T
tangled
Commit e5fb5cd0 authored by ssid's avatar ssid Committed by Commit bot
Browse files
Options

Reland of [tracing] Dump child processes' memory metrics in browser (patchset... 

Reland of [tracing] Dump child processes' memory metrics in browser (patchset #1 id:1 of https://codereview.chromium.org/1617263002/ )

Reason for revert:
The crash was fixed in crrev.com/1620783002

Original issue's description:
> Revert of [tracing] Dump child processes' memory metrics in browser (patchset #1 id:1 of https://codereview.chromium.org/1591553002/ )
>
> Reason for revert:
> TracingBrowserTest.TestMemoryInfra is failing on DrMemory bot with the following
> error:
>
> UNADDRESSABLE ACCESS: reading 0x00000048-0x0000004c 4 byte(s)
> # 0 base.dll!base::Thread::task_runner
> [base\threading\thread.h:168]
> # 1 base.dll!base::trace_event::MemoryDumpManager::CreateProcessDump
> [base\trace_event\memory_dump_manager.cc:313]
> # 2 content.dll!content::TracingControllerImpl::RequestGlobalMemoryDump
> [content\browser\tracing\tracing_controller_impl.cc:1039]
> # 3 content.dll!base::internal::Invoker<>::Run
> [base\bind_internal.h:350]
>
> bug: crbug.com/580295
>
> Original issue's description:
> > Reland of [tracing] Dump child processes' memory metrics in browser (patchset #1 id:1 of https://codereview.chromium.org/1586893003/ )
> >
> > The failure was due to flakiness fixed by crrev.com/1606983002.
> > See crbug.com/578128.
> >
> > Original issue's description:
> > > Revert of [tracing] Dump child processes' memory metrics in browser (patchset #15 id:400001 of https://codereview.chromium.org/1417003003/ )
> > >
> > > Reason for revert:
> > > This is suspected by Findit to cause failure in TracingBrowserTest.TestMemoryInfra under Mac ASan 64 Tests (1).
> > >
> > > Original issue's description:
> > > > [tracing] Dump child processes' memory metrics in browser
> > > >
> > > > The sandbox in linux prevents the process from reading the process
> > > > metrics file. To work around this issue, the browser process will now
> > > > dump statistics of the child processes too. This requires the
> > > > composable dumps support in trace viewer and telemetry.
> > > >
> > > > This CL makes following changes.
> > > > 1. Move the process totals and memory maps dump provider into a single
> > > > header as process_metrics_dump_provider in components/tracing. This is
> > > > because it is not necessary to have this in base and now the provider
> > > > knows/manages for different processes. Also the dump method are made to
> > > > handle error better, since process can vanish while dumping.
> > > >
> > > > 2. Make the dump provider non-singleton and have a register / unregister
> > > > that manages the lifetime of the dump providers.
> > > >
> > > > 3. The dump providers unregister using the new
> > > > UnregisterAndDeleteDumpProviderAsync api added by crrev.com/1430073002.
> > > >
> > > > 4. On linux the browser process dumps metrics for all processes and on
> > > > android the child processes can dump since seccomp sandbox is not
> > > > enabled in android yet.
> > > >
> > > > 5. The proc/status file is human readable stats and is not guaranteed to
> > > > have a field that is asked for. So, the NOTREACHED is removed in
> > > > ReadProcStatusAndGetFieldAsSizeT.
> > > >
> > > > 6. Since we introduce other process dumps from browser process there
> > > > could be races while unregistering and dumping. To test this, the
> > > > browser test is updated.
> > > >
> > > > BUG=461788
> > > >
> > > > Committed: https://crrev.com/4d77d76a42425282b1a3c5b7309db9b98e777f60
> > > > Cr-Commit-Position: refs/heads/master@{#369482}
> > >
> > > TBR=primiano@chromium.org,thakis@chromium.org,simonhatch@chromium.org,sievers@chromium.org,ssid@chromium.org
> > > # Skipping CQ checks because original CL landed less than 1 days ago.
> > > NOPRESUBMIT=true
> > > NOTREECHECKS=true
> > > NOTRY=true
> > > BUG=461788
> > >
> > > Committed: https://crrev.com/93aa967cfcb3e933000f169b9a4f7ac84dbd96da
> > > Cr-Commit-Position: refs/heads/master@{#369535}
> >
> > TBR=primiano@chromium.org,thakis@chromium.org,simonhatch@chromium.org,sievers@chromium.org,huangs@chromium.org
> > BUG=461788
> > CQ_INCLUDE_TRYBOTS=tryserver.blink:linux_blink_rel
> >
> > Committed: https://crrev.com/df4156349708e21844209290a2d3638b550e18b2
> > Cr-Commit-Position: refs/heads/master@{#370195}
>
> TBR=primiano@chromium.org,thakis@chromium.org,simonhatch@chromium.org,sievers@chromium.org,huangs@chromium.org,ssid@chromium.org
> # Not skipping CQ checks because original CL landed more than 1 days ago.
> BUG=461788
>
> Committed: https://crrev.com/94bef8d8bf199e23102783a55a15c9f10bc88631
> Cr-Commit-Position: refs/heads/master@{#370836}

TBR=primiano@chromium.org,thakis@chromium.org,simonhatch@chromium.org,sievers@chromium.org,huangs@chromium.org,oshima@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=461788

Review URL: https://codereview.chromium.org/1629393003

Cr-Commit-Position: refs/heads/master@{#371495}
parent 99a0ad7e
Show whitespace changes
Inline Side-by-side
Showing with 491 additions and 296 deletions
base/BUILD.gn
View file @ e5fb5cd0
......@@ -849,11 +849,8 @@ component("base") {
"trace_event/process_memory_dump.h",
"trace_event/process_memory_maps.cc",
"trace_event/process_memory_maps.h",
"trace_event/process_memory_maps_dump_provider.h",
"trace_event/process_memory_totals.cc",
"trace_event/process_memory_totals.h",
"trace_event/process_memory_totals_dump_provider.cc",
"trace_event/process_memory_totals_dump_provider.h",
"trace_event/trace_buffer.cc",
"trace_event/trace_buffer.h",
"trace_event/trace_config.cc",
......@@ -994,7 +991,6 @@ component("base") {
"sys_info_linux.cc",
"trace_event/malloc_dump_provider.cc",
"trace_event/malloc_dump_provider.h",
"trace_event/process_memory_maps_dump_provider.cc",
]
set_sources_assignment_filter(sources_assignment_filter)
......@@ -1061,7 +1057,6 @@ component("base") {
"sync_socket_posix.cc",
"sys_info.cc",
"sys_info_posix.cc",
"trace_event/process_memory_totals_dump_provider.cc",
"trace_event/trace_event_system_stats_monitor.cc",
]
......@@ -1190,7 +1185,6 @@ component("base") {
sources += [
"trace_event/malloc_dump_provider.cc",
"trace_event/malloc_dump_provider.h",
"trace_event/process_memory_maps_dump_provider.cc",
]
if (is_asan || is_lsan || is_msan || is_tsan) {
......@@ -1828,7 +1822,6 @@ test("base_unittests") {
"trace_event/memory_allocator_dump_unittest.cc",
"trace_event/memory_dump_manager_unittest.cc",
"trace_event/process_memory_dump_unittest.cc",
"trace_event/process_memory_totals_dump_provider_unittest.cc",
"trace_event/trace_config_memory_test_util.h",
"trace_event/trace_config_unittest.cc",
"trace_event/trace_event_argument_unittest.cc",
......@@ -1945,10 +1938,6 @@ test("base_unittests") {
}
}
if (is_linux || is_android) {
sources += [ "trace_event/process_memory_maps_dump_provider_unittest.cc" ]
}
if (!is_linux || use_ozone) {
sources -= [ "message_loop/message_pump_glib_unittest.cc" ]
}
......
base/process/process_metrics_linux.cc
View file @ e5fb5cd0
......@@ -86,7 +86,8 @@ size_t ReadProcStatusAndGetFieldAsSizeT(pid_t pid, const std::string& field) {
return value;
}
}
NOTREACHED();
// This can be reached if the process dies when proc is read -- in that case,
// the kernel can return missing fields.
return 0;
}
......
base/trace_event/memory_dump_manager.cc
View file @ e5fb5cd0
......@@ -23,14 +23,6 @@
#include "base/trace_event/trace_event_argument.h"
#include "build/build_config.h"
#if !defined(OS_NACL)
#include "base/trace_event/process_memory_totals_dump_provider.h"
#endif
#if defined(OS_LINUX) || defined(OS_ANDROID)
#include "base/trace_event/process_memory_maps_dump_provider.h"
#endif
#if defined(OS_ANDROID)
#include "base/trace_event/java_heap_dump_provider_android.h"
#endif
......@@ -166,20 +158,10 @@ void MemoryDumpManager::Initialize(MemoryDumpManagerDelegate* delegate,
}
// Enable the core dump providers.
#if !defined(OS_NACL)
RegisterDumpProvider(ProcessMemoryTotalsDumpProvider::GetInstance(),
"ProcessMemoryTotals", nullptr);
#endif
#if defined(MALLOC_MEMORY_TRACING_SUPPORTED)
RegisterDumpProvider(MallocDumpProvider::GetInstance(), "Malloc", nullptr);
#endif
#if defined(OS_LINUX) || defined(OS_ANDROID)
RegisterDumpProvider(ProcessMemoryMapsDumpProvider::GetInstance(),
"ProcessMemoryMaps", nullptr);
#endif
#if defined(OS_ANDROID)
RegisterDumpProvider(JavaHeapDumpProvider::GetInstance(), "JavaHeap",
nullptr);
......
base/trace_event/process_memory_maps_dump_provider.h deleted 100644 → 0
View file @ 99a0ad7e
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_TRACE_EVENT_PROCESS_MEMORY_MAPS_DUMP_PROVIDER_H_
#define BASE_TRACE_EVENT_PROCESS_MEMORY_MAPS_DUMP_PROVIDER_H_
#include "base/gtest_prod_util.h"
#include "base/macros.h"
#include "base/memory/singleton.h"
#include "base/trace_event/memory_dump_provider.h"
#include "build/build_config.h"
namespace base {
namespace trace_event {
// Dump provider which collects process-wide memory stats.
class BASE_EXPORT ProcessMemoryMapsDumpProvider : public MemoryDumpProvider {
public:
static ProcessMemoryMapsDumpProvider* GetInstance();
// MemoryDumpProvider implementation.
bool OnMemoryDump(const MemoryDumpArgs& args,
ProcessMemoryDump* pmd) override;
private:
friend struct DefaultSingletonTraits<ProcessMemoryMapsDumpProvider>;
FRIEND_TEST_ALL_PREFIXES(ProcessMemoryMapsDumpProviderTest, ParseProcSmaps);
#if defined(OS_LINUX) || defined(OS_ANDROID) || defined(OS_NACL)
static FILE* proc_smaps_for_testing;
#endif
ProcessMemoryMapsDumpProvider();
~ProcessMemoryMapsDumpProvider() override;
DISALLOW_COPY_AND_ASSIGN(ProcessMemoryMapsDumpProvider);
};
} // namespace trace_event
} // namespace base
#endif // BASE_TRACE_EVENT_PROCESS_MEMORY_MAPS_DUMP_PROVIDER_H_
base/trace_event/process_memory_totals_dump_provider.cc deleted 100644 → 0
View file @ 99a0ad7e
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/trace_event/process_memory_totals_dump_provider.h"
#include <stddef.h>
#include "base/process/process_metrics.h"
#include "base/trace_event/process_memory_dump.h"
#include "base/trace_event/process_memory_totals.h"
#include "build/build_config.h"
#if defined(OS_LINUX) || defined(OS_ANDROID)
#include <fcntl.h>
#include "base/files/file_util.h"
namespace {
bool kernel_supports_rss_peak_reset = true;
const char kClearPeakRssCommand[] = "5";
}
#endif
namespace base {
namespace trace_event {
// static
uint64_t ProcessMemoryTotalsDumpProvider::rss_bytes_for_testing = 0;
// static
ProcessMemoryTotalsDumpProvider*
ProcessMemoryTotalsDumpProvider::GetInstance() {
return Singleton<
ProcessMemoryTotalsDumpProvider,
LeakySingletonTraits<ProcessMemoryTotalsDumpProvider>>::get();
}
ProcessMemoryTotalsDumpProvider::ProcessMemoryTotalsDumpProvider()
: process_metrics_(ProcessMetrics::CreateCurrentProcessMetrics()) {}
ProcessMemoryTotalsDumpProvider::~ProcessMemoryTotalsDumpProvider() {
}
// Called at trace dump point time. Creates a snapshot the memory counters for
// the current process.
bool ProcessMemoryTotalsDumpProvider::OnMemoryDump(const MemoryDumpArgs& args,
ProcessMemoryDump* pmd) {
const uint64_t rss_bytes = rss_bytes_for_testing
? rss_bytes_for_testing
: process_metrics_->GetWorkingSetSize();
uint64_t peak_rss_bytes = 0;
#if !defined(OS_IOS)
peak_rss_bytes = process_metrics_->GetPeakWorkingSetSize();
#if defined(OS_LINUX) || defined(OS_ANDROID)
if (kernel_supports_rss_peak_reset) {
// TODO(ssid): Fix crbug.com/461788 to write to the file from sandboxed
// processes.
int clear_refs_fd = open("/proc/self/clear_refs", O_WRONLY);
if (clear_refs_fd > 0 &&
WriteFileDescriptor(clear_refs_fd, kClearPeakRssCommand,
sizeof(kClearPeakRssCommand))) {
pmd->process_totals()->set_is_peak_rss_resetable(true);
} else {
kernel_supports_rss_peak_reset = false;
}
close(clear_refs_fd);
}
#elif defined(OS_MACOSX)
size_t private_bytes;
bool res = process_metrics_->GetMemoryBytes(&private_bytes,
nullptr /* shared_bytes */);
if (res) {
pmd->process_totals()->SetExtraFieldInBytes("private_bytes", private_bytes);
}
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
#endif // !defined(OS_IOS)
if (rss_bytes > 0) {
pmd->process_totals()->set_resident_set_bytes(rss_bytes);
pmd->process_totals()->set_peak_resident_set_bytes(peak_rss_bytes);
pmd->set_has_process_totals();
return true;
}
return false;
}
} // namespace trace_event
} // namespace base
base/trace_event/process_memory_totals_dump_provider.h deleted 100644 → 0
View file @ 99a0ad7e
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_TRACE_EVENT_PROCESS_MEMORY_TOTALS_DUMP_PROVIDER_H_
#define BASE_TRACE_EVENT_PROCESS_MEMORY_TOTALS_DUMP_PROVIDER_H_
#include <stdint.h>
#include "base/gtest_prod_util.h"
#include "base/macros.h"
#include "base/memory/scoped_ptr.h"
#include "base/memory/singleton.h"
#include "base/trace_event/memory_dump_provider.h"
namespace base {
class ProcessMetrics;
namespace trace_event {
// Dump provider which collects process-wide memory stats.
class BASE_EXPORT ProcessMemoryTotalsDumpProvider : public MemoryDumpProvider {
public:
static ProcessMemoryTotalsDumpProvider* GetInstance();
// MemoryDumpProvider implementation.
bool OnMemoryDump(const MemoryDumpArgs& args,
ProcessMemoryDump* pmd) override;
private:
friend struct DefaultSingletonTraits<ProcessMemoryTotalsDumpProvider>;
FRIEND_TEST_ALL_PREFIXES(ProcessMemoryTotalsDumpProviderTest, DumpRSS);
static uint64_t rss_bytes_for_testing;
ProcessMemoryTotalsDumpProvider();
~ProcessMemoryTotalsDumpProvider() override;
scoped_ptr<ProcessMetrics> process_metrics_;
DISALLOW_COPY_AND_ASSIGN(ProcessMemoryTotalsDumpProvider);
};
} // namespace trace_event
} // namespace base
#endif // BASE_TRACE_EVENT_PROCESS_MEMORY_TOTALS_DUMP_PROVIDER_H_
base/trace_event/process_memory_totals_dump_provider_unittest.cc deleted 100644 → 0
View file @ 99a0ad7e
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/trace_event/process_memory_totals_dump_provider.h"
#include <stddef.h>
#include <stdint.h>
#include "base/trace_event/process_memory_dump.h"
#include "base/trace_event/process_memory_totals.h"
#include "testing/gtest/include/gtest/gtest.h"
namespace base {
namespace trace_event {
TEST(ProcessMemoryTotalsDumpProviderTest, DumpRSS) {
const MemoryDumpArgs high_detail_args = {MemoryDumpLevelOfDetail::DETAILED};
auto pmtdp = ProcessMemoryTotalsDumpProvider::GetInstance();
scoped_ptr<ProcessMemoryDump> pmd_before(new ProcessMemoryDump(nullptr));
scoped_ptr<ProcessMemoryDump> pmd_after(new ProcessMemoryDump(nullptr));
ProcessMemoryTotalsDumpProvider::rss_bytes_for_testing = 1024;
pmtdp->OnMemoryDump(high_detail_args, pmd_before.get());
// Pretend that the RSS of the process increased of +1M.
const size_t kAllocSize = 1048576;
ProcessMemoryTotalsDumpProvider::rss_bytes_for_testing += kAllocSize;
pmtdp->OnMemoryDump(high_detail_args, pmd_after.get());
ProcessMemoryTotalsDumpProvider::rss_bytes_for_testing = 0;
ASSERT_TRUE(pmd_before->has_process_totals());
ASSERT_TRUE(pmd_after->has_process_totals());
const uint64_t rss_before =
pmd_before->process_totals()->resident_set_bytes();
const uint64_t rss_after = pmd_after->process_totals()->resident_set_bytes();
EXPECT_NE(0U, rss_before);
EXPECT_NE(0U, rss_after);
EXPECT_EQ(rss_after - rss_before, kAllocSize);
}
} // namespace trace_event
} // namespace base
base/trace_event/trace_event.gypi
View file @ e5fb5cd0
......@@ -36,11 +36,8 @@
'trace_event/process_memory_dump.h',
'trace_event/process_memory_maps.cc',
'trace_event/process_memory_maps.h',
'trace_event/process_memory_maps_dump_provider.h',
'trace_event/process_memory_totals.cc',
'trace_event/process_memory_totals.h',
'trace_event/process_memory_totals_dump_provider.cc',
'trace_event/process_memory_totals_dump_provider.h',
'trace_event/trace_buffer.cc',
'trace_event/trace_buffer.h',
'trace_event/trace_config.cc',
......@@ -79,7 +76,6 @@
'trace_event/memory_allocator_dump_unittest.cc',
'trace_event/memory_dump_manager_unittest.cc',
'trace_event/process_memory_dump_unittest.cc',
'trace_event/process_memory_totals_dump_provider_unittest.cc',
'trace_event/trace_config_memory_test_util.h',
'trace_event/trace_config_unittest.cc',
'trace_event/trace_event_argument_unittest.cc',
......@@ -95,14 +91,6 @@
'trace_event/malloc_dump_provider.h',
],
}],
['OS == "linux" or OS == "android"', {
'trace_event_sources': [
'trace_event/process_memory_maps_dump_provider.cc',
],
'trace_event_test_sources' : [
'trace_event/process_memory_maps_dump_provider_unittest.cc',
],
}],
['OS == "android"', {
'trace_event_test_sources' : [
'trace_event/trace_event_android_unittest.cc',
......
chrome/test/base/tracing_browsertest.cc
View file @ e5fb5cd0
......@@ -61,12 +61,22 @@ class TracingBrowserTest : public InProcessBrowserTest {
MemoryDumpManager::kTraceCategory,
event_name, 10));
GURL url2("chrome://credits/");
// Create and destroy renderers while tracing is enabled.
GURL url2("chrome://credits");
ui_test_utils::NavigateToURLWithDisposition(
browser(), url2, NEW_FOREGROUND_TAB,
ui_test_utils::BROWSER_TEST_WAIT_FOR_NAVIGATION);
ASSERT_NO_FATAL_FAILURE(ExecuteJavascriptOnCurrentTab());
// Close the current tab.
browser()->tab_strip_model()->CloseSelectedTabs();
GURL url3("chrome://settings");
ui_test_utils::NavigateToURLWithDisposition(
browser(), url3, CURRENT_TAB,
ui_test_utils::BROWSER_TEST_WAIT_FOR_NAVIGATION);
ASSERT_NO_FATAL_FAILURE(ExecuteJavascriptOnCurrentTab());
EXPECT_TRUE(WaitForWatchEvent(no_timeout));
ASSERT_TRUE(EndTracing(&json_events));
......
components/components_tests.gyp
View file @ e5fb5cd0
......@@ -758,6 +758,7 @@
],
'tracing_unittest_sources': [
'tracing/graphics_memory_dump_provider_android_unittest.cc',
'tracing/process_metrics_memory_dump_provider_unittest.cc',
'tracing/trace_config_file_unittest.cc',
],
'translate_unittest_sources': [
......
components/tracing.gyp
View file @ e5fb5cd0
......@@ -32,6 +32,8 @@
'tracing/child_trace_message_filter.h',
'tracing/graphics_memory_dump_provider_android.cc',
'tracing/graphics_memory_dump_provider_android.h',
'tracing/process_metrics_memory_dump_provider.cc',
'tracing/process_metrics_memory_dump_provider.h',
'tracing/trace_config_file.cc',
'tracing/trace_config_file.h',
'tracing/trace_to_console.cc',
......@@ -42,6 +44,13 @@
'tracing/tracing_switches.cc',
'tracing/tracing_switches.h',
],
'target_conditions': [
['>(nacl_untrusted_build)==1', {
'sources!': [
'tracing/process_metrics_memory_dump_provider.cc',
],
}],
]
},
],
}
components/tracing/BUILD.gn
View file @ e5fb5cd0
......@@ -10,6 +10,8 @@ component("tracing") {
"child_trace_message_filter.h",
"graphics_memory_dump_provider_android.cc",
"graphics_memory_dump_provider_android.h",
"process_metrics_memory_dump_provider.cc",
"process_metrics_memory_dump_provider.h",
"tracing_export.h",
"tracing_messages.cc",
"tracing_messages.h",
......@@ -21,6 +23,10 @@ component("tracing") {
"//base",
"//ipc",
]
if (is_nacl) {
sources -= [ "process_metrics_memory_dump_provider.cc" ]
}
}
component("startup_tracing") {
......@@ -46,6 +52,7 @@ source_set("unit_tests") {
sources = [
"graphics_memory_dump_provider_android_unittest.cc",
"process_metrics_memory_dump_provider_unittest.cc",
]
deps = [
......
components/tracing/child_memory_dump_manager_delegate_impl.cc
View file @ e5fb5cd0
......@@ -5,7 +5,9 @@
#include "components/tracing/child_memory_dump_manager_delegate_impl.h"
#include "base/single_thread_task_runner.h"
#include "build/build_config.h"
#include "components/tracing/child_trace_message_filter.h"
#include "components/tracing/process_metrics_memory_dump_provider.h"
namespace tracing {
......@@ -49,6 +51,13 @@ void ChildMemoryDumpManagerDelegateImpl::SetChildTraceMessageFilter(
if (ctmf) {
base::trace_event::MemoryDumpManager::GetInstance()->Initialize(
this /* delegate */, false /* is_coordinator */);
#if !defined(OS_LINUX) && !defined(OS_NACL)
// On linux the browser process takes care of dumping process metrics.
// The child process is not allowed to do so due to BPF sandbox.
tracing::ProcessMetricsMemoryDumpProvider::RegisterForProcess(
base::kNullProcessId);
#endif
}
}
......
base/trace_event/process_memory_maps_dump_provider.cc → components/tracing/process_metrics_memory_dump_provider.cc
View file @ e5fb5cd0
......@@ -2,29 +2,42 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/trace_event/process_memory_maps_dump_provider.h"
#include "components/tracing/process_metrics_memory_dump_provider.h"
#include <fcntl.h>
#include <stdint.h>
#include <map>
#include "base/files/file_util.h"
#include "base/files/scoped_file.h"
#include "base/format_macros.h"
#include "base/lazy_instance.h"
#include "base/logging.h"
#include "base/process/process_metrics.h"
#include "base/strings/string_number_conversions.h"
#include "base/strings/string_util.h"
#include "base/trace_event/memory_dump_manager.h"
#include "base/trace_event/process_memory_dump.h"
#include "base/trace_event/process_memory_maps.h"
#include "base/trace_event/process_memory_totals.h"
#include "build/build_config.h"
namespace base {
namespace trace_event {
// static
FILE* ProcessMemoryMapsDumpProvider::proc_smaps_for_testing = nullptr;
namespace tracing {
namespace {
base::LazyInstance<
std::map<base::ProcessId, scoped_ptr<ProcessMetricsMemoryDumpProvider>>>::
Leaky g_dump_providers_map = LAZY_INSTANCE_INITIALIZER;
#if defined(OS_LINUX) || defined(OS_ANDROID)
const char kClearPeakRssCommand[] = "5";
const uint32_t kMaxLineSize = 4096;
bool ParseSmapsHeader(const char* header_line,
ProcessMemoryMaps::VMRegion* region) {
base::trace_event::ProcessMemoryMaps::VMRegion* region) {
// e.g., "00400000-00421000 r-xp 00000000 fc:01 1234 /foo.so\n"
bool res = true; // Whether this region should be appended or skipped.
uint64_t end_addr = 0;
......@@ -47,19 +60,20 @@ bool ParseSmapsHeader(const char* header_line,
region->protection_flags = 0;
if (protection_flags[0] == 'r') {
region->protection_flags |=
ProcessMemoryMaps::VMRegion::kProtectionFlagsRead;
base::trace_event::ProcessMemoryMaps::VMRegion::kProtectionFlagsRead;
}
if (protection_flags[1] == 'w') {
region->protection_flags |=
ProcessMemoryMaps::VMRegion::kProtectionFlagsWrite;
base::trace_event::ProcessMemoryMaps::VMRegion::kProtectionFlagsWrite;
}
if (protection_flags[2] == 'x') {
region->protection_flags |=
ProcessMemoryMaps::VMRegion::kProtectionFlagsExec;
base::trace_event::ProcessMemoryMaps::VMRegion::kProtectionFlagsExec;
}
region->mapped_file = mapped_file;
TrimWhitespaceASCII(region->mapped_file, TRIM_ALL, &region->mapped_file);
base::TrimWhitespaceASCII(region->mapped_file, base::TRIM_ALL,
&region->mapped_file);
return res;
}
......@@ -67,17 +81,18 @@ bool ParseSmapsHeader(const char* header_line,
uint64_t ReadCounterBytes(char* counter_line) {
uint64_t counter_value = 0;
int res = sscanf(counter_line, "%*s %" SCNu64 " kB", &counter_value);
DCHECK_EQ(1, res);
return counter_value * 1024;
return res == 1 ? counter_value * 1024 : 0;
}
uint32_t ParseSmapsCounter(char* counter_line,
ProcessMemoryMaps::VMRegion* region) {
uint32_t ParseSmapsCounter(
char* counter_line,
base::trace_event::ProcessMemoryMaps::VMRegion* region) {
// A smaps counter lines looks as follows: "RSS: 0 Kb\n"
uint32_t res = 1;
char counter_name[20];
int did_read = sscanf(counter_line, "%19[^\n ]", counter_name);
DCHECK_EQ(1, did_read);
if (did_read != 1)
return 0;
if (strcmp(counter_name, "Pss:") == 0) {
region->byte_stats_proportional_resident = ReadCounterBytes(counter_line);
......@@ -98,7 +113,8 @@ uint32_t ParseSmapsCounter(char* counter_line,
return res;
}
uint32_t ReadLinuxProcSmapsFile(FILE* smaps_file, ProcessMemoryMaps* pmm) {
uint32_t ReadLinuxProcSmapsFile(FILE* smaps_file,
base::trace_event::ProcessMemoryMaps* pmm) {
if (!smaps_file)
return 0;
......@@ -108,15 +124,14 @@ uint32_t ReadLinuxProcSmapsFile(FILE* smaps_file, ProcessMemoryMaps* pmm) {
const uint32_t kNumExpectedCountersPerRegion = 6;
uint32_t counters_parsed_for_current_region = 0;
uint32_t num_valid_regions = 0;
ProcessMemoryMaps::VMRegion region;
base::trace_event::ProcessMemoryMaps::VMRegion region;
bool should_add_current_region = false;
for (;;) {
line[0] = '\0';
if (fgets(line, kMaxLineSize, smaps_file) == nullptr)
if (fgets(line, kMaxLineSize, smaps_file) == nullptr || !strlen(line))
break;
DCHECK_GT(strlen(line), 0u);
if (isxdigit(line[0]) && !isupper(line[0])) {
region = ProcessMemoryMaps::VMRegion();
region = base::trace_event::ProcessMemoryMaps::VMRegion();
counters_parsed_for_current_region = 0;
should_add_current_region = ParseSmapsHeader(line, &region);
} else {
......@@ -134,43 +149,157 @@ uint32_t ReadLinuxProcSmapsFile(FILE* smaps_file, ProcessMemoryMaps* pmm) {
}
return num_valid_regions;
}
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
scoped_ptr<base::ProcessMetrics> CreateProcessMetrics(base::ProcessId process) {
if (process == base::kNullProcessId)
return make_scoped_ptr(base::ProcessMetrics::CreateCurrentProcessMetrics());
#if defined(OS_LINUX) || defined(OS_ANDROID)
// Just pass ProcessId instead of handle since they are the same in linux and
// android.
return make_scoped_ptr(base::ProcessMetrics::CreateProcessMetrics(process));
#else
// Creating process metrics for child processes in mac or windows requires
// additional information like ProcessHandle or port provider. This is a non
// needed use case.
NOTREACHED();
return scoped_ptr<base::ProcessMetrics>();
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
}
} // namespace
// static
ProcessMemoryMapsDumpProvider* ProcessMemoryMapsDumpProvider::GetInstance() {
return Singleton<ProcessMemoryMapsDumpProvider,
LeakySingletonTraits<ProcessMemoryMapsDumpProvider>>::get();
uint64_t ProcessMetricsMemoryDumpProvider::rss_bytes_for_testing = 0;
#if defined(OS_LINUX) || defined(OS_ANDROID)
// static
FILE* ProcessMetricsMemoryDumpProvider::proc_smaps_for_testing = nullptr;
bool ProcessMetricsMemoryDumpProvider::DumpProcessMemoryMaps(
const base::trace_event::MemoryDumpArgs& args,
base::trace_event::ProcessMemoryDump* pmd) {
uint32_t res = 0;
if (proc_smaps_for_testing) {
res = ReadLinuxProcSmapsFile(proc_smaps_for_testing, pmd->process_mmaps());
} else {
std::string file_name = "/proc/" + (process_ == base::kNullProcessId
? "self"
: base::IntToString(process_)) +
"/smaps";
base::ScopedFILE smaps_file(fopen(file_name.c_str(), "r"));
res = ReadLinuxProcSmapsFile(smaps_file.get(), pmd->process_mmaps());
}
if (res)
pmd->set_has_process_mmaps();
return res;
}
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
ProcessMemoryMapsDumpProvider::ProcessMemoryMapsDumpProvider() {
// static
void ProcessMetricsMemoryDumpProvider::RegisterForProcess(
base::ProcessId process) {
scoped_ptr<ProcessMetricsMemoryDumpProvider> metrics_provider(
new ProcessMetricsMemoryDumpProvider(process));
base::trace_event::MemoryDumpProvider::Options options(process);
base::trace_event::MemoryDumpManager::GetInstance()->RegisterDumpProvider(
metrics_provider.get(), "ProcessMemoryMetrics", nullptr, options);
bool did_insert =
g_dump_providers_map.Get()
.insert(std::make_pair(process, std::move(metrics_provider)))
.second;
if (!did_insert) {
DLOG(ERROR) << "ProcessMetricsMemoryDumpProvider already registered for "
<< (process == base::kNullProcessId
? "current process"
: "process id " + base::IntToString(process));
}
}
ProcessMemoryMapsDumpProvider::~ProcessMemoryMapsDumpProvider() {
// static
void ProcessMetricsMemoryDumpProvider::UnregisterForProcess(
base::ProcessId process) {
auto iter = g_dump_providers_map.Get().find(process);
if (iter == g_dump_providers_map.Get().end()) {
return;
}
base::trace_event::MemoryDumpManager::GetInstance()
->UnregisterAndDeleteDumpProviderSoon(std::move(iter->second));
g_dump_providers_map.Get().erase(iter);
}
ProcessMetricsMemoryDumpProvider::ProcessMetricsMemoryDumpProvider(
base::ProcessId process)
: process_(process),
process_metrics_(CreateProcessMetrics(process)),
is_rss_peak_resettable_(true) {}
ProcessMetricsMemoryDumpProvider::~ProcessMetricsMemoryDumpProvider() {}
// Called at trace dump point time. Creates a snapshot of the memory maps for
// the current process.
bool ProcessMemoryMapsDumpProvider::OnMemoryDump(const MemoryDumpArgs& args,
ProcessMemoryDump* pmd) {
// Snapshot of memory maps is not taken for light dump requests.
if (args.level_of_detail == MemoryDumpLevelOfDetail::LIGHT)
return true;
bool ProcessMetricsMemoryDumpProvider::OnMemoryDump(
const base::trace_event::MemoryDumpArgs& args,
base::trace_event::ProcessMemoryDump* pmd) {
bool res = DumpProcessTotals(args, pmd);
uint32_t res = 0;
if (UNLIKELY(proc_smaps_for_testing)) {
res = ReadLinuxProcSmapsFile(proc_smaps_for_testing, pmd->process_mmaps());
#if defined(OS_LINUX) || defined(OS_ANDROID)
if (args.level_of_detail ==
base::trace_event::MemoryDumpLevelOfDetail::DETAILED)
res &= DumpProcessMemoryMaps(args, pmd);
#endif
return res;
}
bool ProcessMetricsMemoryDumpProvider::DumpProcessTotals(
const base::trace_event::MemoryDumpArgs& args,
base::trace_event::ProcessMemoryDump* pmd) {
const uint64_t rss_bytes = rss_bytes_for_testing
? rss_bytes_for_testing
: process_metrics_->GetWorkingSetSize();
// rss_bytes will be 0 if the process ended while dumping.
if (!rss_bytes)
return false;
uint64_t peak_rss_bytes = 0;
#if !defined(OS_IOS)
peak_rss_bytes = process_metrics_->GetPeakWorkingSetSize();
#if defined(OS_LINUX) || defined(OS_ANDROID)
if (is_rss_peak_resettable_) {
std::string clear_refs_file =
"/proc/" +
(process_ == base::kNullProcessId ? "self"
: base::IntToString(process_)) +
"/clear_refs";
int clear_refs_fd = open(clear_refs_file.c_str(), O_WRONLY);
if (clear_refs_fd > 0 &&
base::WriteFileDescriptor(clear_refs_fd, kClearPeakRssCommand,
sizeof(kClearPeakRssCommand))) {
pmd->process_totals()->set_is_peak_rss_resetable(true);
} else {
ScopedFILE smaps_file(fopen("/proc/self/smaps", "r"));
res = ReadLinuxProcSmapsFile(smaps_file.get(), pmd->process_mmaps());
is_rss_peak_resettable_ = false;
}
close(clear_refs_fd);
}
#elif defined(MACOSX)
size_t private_bytes;
bool res = process_metrics_->GetMemoryBytes(&private_bytes,
nullptr /* shared_bytes */);
if (res)
pmd->process_totals()->SetExtraFieldInBytes("private_bytes", private_bytes);
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
#endif // !defined(OS_IOS)
if (res > 0) {
pmd->set_has_process_mmaps();
pmd->process_totals()->set_resident_set_bytes(rss_bytes);
pmd->set_has_process_totals();
pmd->process_totals()->set_peak_resident_set_bytes(peak_rss_bytes);
// Returns true even if other metrics failed, since rss is reported.
return true;
}
return false;
}
} // namespace trace_event
} // namespace base
} // namespace tracing
components/tracing/process_metrics_memory_dump_provider.h 0 → 100644
View file @ e5fb5cd0
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef COMPONENTS_TRACING_PROCESS_MEMORY_METRICS_DUMP_PROVIDER_H_
#define COMPONENTS_TRACING_PROCESS_MEMORY_METRICS_DUMP_PROVIDER_H_
#include "base/gtest_prod_util.h"
#include "base/macros.h"
#include "base/memory/scoped_ptr.h"
#include "base/process/process_handle.h"
#include "base/trace_event/memory_dump_provider.h"
#include "build/build_config.h"
#include "components/tracing/tracing_export.h"
namespace base {
class ProcessMetrics;
}
namespace tracing {
// Dump provider which collects process-wide memory stats.
class TRACING_EXPORT ProcessMetricsMemoryDumpProvider
: public base::trace_event::MemoryDumpProvider {
public:
// Pass base::kNullProcessId to register for current process.
static void RegisterForProcess(base::ProcessId process);
static void UnregisterForProcess(base::ProcessId process);
~ProcessMetricsMemoryDumpProvider() override;
// MemoryDumpProvider implementation.
bool OnMemoryDump(const base::trace_event::MemoryDumpArgs& args,
base::trace_event::ProcessMemoryDump* pmd) override;
private:
FRIEND_TEST_ALL_PREFIXES(ProcessMetricsMemoryDumpProviderTest,
ParseProcSmaps);
FRIEND_TEST_ALL_PREFIXES(ProcessMetricsMemoryDumpProviderTest, DumpRSS);
ProcessMetricsMemoryDumpProvider(base::ProcessId process);
bool DumpProcessTotals(const base::trace_event::MemoryDumpArgs& args,
base::trace_event::ProcessMemoryDump* pmd);
bool DumpProcessMemoryMaps(const base::trace_event::MemoryDumpArgs& args,
base::trace_event::ProcessMemoryDump* pmd);
static uint64_t rss_bytes_for_testing;
#if defined(OS_LINUX) || defined(OS_ANDROID)
static FILE* proc_smaps_for_testing;
#endif
base::ProcessId process_;
scoped_ptr<base::ProcessMetrics> process_metrics_;
// The peak may not be resettable on all the processes if the linux kernel is
// older than http://bit.ly/reset_rss or only on child processes if yama LSM
// sandbox is enabled.
bool is_rss_peak_resettable_;
DISALLOW_COPY_AND_ASSIGN(ProcessMetricsMemoryDumpProvider);
};
} // namespace tracing
#endif // COMPONENTS_TRACING_PROCESS_MEMORY_METRICS_DUMP_PROVIDER_H_
base/trace_event/process_memory_maps_dump_provider_unittest.cc → components/tracing/process_metrics_memory_dump_provider_unittest.cc
View file @ e5fb5cd0
......@@ -2,19 +2,20 @@
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/trace_event/process_memory_maps_dump_provider.h"
#include "components/tracing/process_metrics_memory_dump_provider.h"
#include <stdint.h>
#include "base/files/file_util.h"
#include "base/trace_event/process_memory_dump.h"
#include "base/trace_event/process_memory_maps.h"
#include "base/trace_event/process_memory_totals.h"
#include "base/trace_event/trace_event_argument.h"
#include "testing/gtest/include/gtest/gtest.h"
namespace base {
namespace trace_event {
namespace tracing {
#if defined(OS_LINUX) || defined(OS_ANDROID)
namespace {
const char kTestSmaps1[] =
"00400000-004be000 r-xp 00000000 fc:01 1234 /file/1\n"
......@@ -105,8 +106,8 @@ const char kTestSmaps2[] =
"VmFlags: rd wr mr mw me ac sd\n";
void CreateAndSetSmapsFileForTesting(const char* smaps_string,
ScopedFILE& file) {
FilePath temp_path;
base::ScopedFILE& file) {
base::FilePath temp_path;
FILE* temp_file = CreateAndOpenTemporaryFile(&temp_path);
file.reset(temp_file);
ASSERT_TRUE(temp_file);
......@@ -116,28 +117,69 @@ void CreateAndSetSmapsFileForTesting(const char* smaps_string,
}
} // namespace
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
TEST(ProcessMemoryMapsDumpProviderTest, ParseProcSmaps) {
const uint32_t kProtR = ProcessMemoryMaps::VMRegion::kProtectionFlagsRead;
const uint32_t kProtW = ProcessMemoryMaps::VMRegion::kProtectionFlagsWrite;
const uint32_t kProtX = ProcessMemoryMaps::VMRegion::kProtectionFlagsExec;
const MemoryDumpArgs dump_args = {MemoryDumpLevelOfDetail::DETAILED};
TEST(ProcessMetricsMemoryDumpProviderTest, DumpRSS) {
const base::trace_event::MemoryDumpArgs high_detail_args = {
base::trace_event::MemoryDumpLevelOfDetail::DETAILED};
scoped_ptr<ProcessMetricsMemoryDumpProvider> pmtdp(
new ProcessMetricsMemoryDumpProvider(base::kNullProcessId));
scoped_ptr<base::trace_event::ProcessMemoryDump> pmd_before(
new base::trace_event::ProcessMemoryDump(nullptr));
scoped_ptr<base::trace_event::ProcessMemoryDump> pmd_after(
new base::trace_event::ProcessMemoryDump(nullptr));
auto pmmdp = ProcessMemoryMapsDumpProvider::GetInstance();
ProcessMetricsMemoryDumpProvider::rss_bytes_for_testing = 1024;
pmtdp->OnMemoryDump(high_detail_args, pmd_before.get());
// Pretend that the RSS of the process increased of +1M.
const size_t kAllocSize = 1048576;
ProcessMetricsMemoryDumpProvider::rss_bytes_for_testing += kAllocSize;
pmtdp->OnMemoryDump(high_detail_args, pmd_after.get());
ProcessMetricsMemoryDumpProvider::rss_bytes_for_testing = 0;
ASSERT_TRUE(pmd_before->has_process_totals());
ASSERT_TRUE(pmd_after->has_process_totals());
const uint64_t rss_before =
pmd_before->process_totals()->resident_set_bytes();
const uint64_t rss_after = pmd_after->process_totals()->resident_set_bytes();
EXPECT_NE(0U, rss_before);
EXPECT_NE(0U, rss_after);
EXPECT_EQ(rss_after - rss_before, kAllocSize);
}
#if defined(OS_LINUX) || defined(OS_ANDROID)
TEST(ProcessMetricsMemoryDumpProviderTest, ParseProcSmaps) {
const uint32_t kProtR =
base::trace_event::ProcessMemoryMaps::VMRegion::kProtectionFlagsRead;
const uint32_t kProtW =
base::trace_event::ProcessMemoryMaps::VMRegion::kProtectionFlagsWrite;
const uint32_t kProtX =
base::trace_event::ProcessMemoryMaps::VMRegion::kProtectionFlagsExec;
const base::trace_event::MemoryDumpArgs dump_args = {
base::trace_event::MemoryDumpLevelOfDetail::DETAILED};
scoped_ptr<ProcessMetricsMemoryDumpProvider> pmmdp(
new ProcessMetricsMemoryDumpProvider(base::kNullProcessId));
// Emulate an empty /proc/self/smaps.
ProcessMemoryDump pmd_invalid(nullptr /* session_state */);
ScopedFILE empty_file(OpenFile(FilePath("/dev/null"), "r"));
base::trace_event::ProcessMemoryDump pmd_invalid(nullptr /* session_state */);
base::ScopedFILE empty_file(OpenFile(base::FilePath("/dev/null"), "r"));
ASSERT_TRUE(empty_file.get());
ProcessMemoryMapsDumpProvider::proc_smaps_for_testing = empty_file.get();
ProcessMetricsMemoryDumpProvider::proc_smaps_for_testing = empty_file.get();
pmmdp->OnMemoryDump(dump_args, &pmd_invalid);
ASSERT_FALSE(pmd_invalid.has_process_mmaps());
// Parse the 1st smaps file.
ProcessMemoryDump pmd_1(nullptr /* session_state */);
ScopedFILE temp_file1;
base::trace_event::ProcessMemoryDump pmd_1(nullptr /* session_state */);
base::ScopedFILE temp_file1;
CreateAndSetSmapsFileForTesting(kTestSmaps1, temp_file1);
ProcessMemoryMapsDumpProvider::proc_smaps_for_testing = temp_file1.get();
ProcessMetricsMemoryDumpProvider::proc_smaps_for_testing = temp_file1.get();
pmmdp->OnMemoryDump(dump_args, &pmd_1);
ASSERT_TRUE(pmd_1.has_process_mmaps());
const auto& regions_1 = pmd_1.process_mmaps()->vm_regions();
......@@ -166,10 +208,10 @@ TEST(ProcessMemoryMapsDumpProviderTest, ParseProcSmaps) {
EXPECT_EQ(0 * 1024UL, regions_1[1].byte_stats_swapped);
// Parse the 2nd smaps file.
ProcessMemoryDump pmd_2(nullptr /* session_state */);
ScopedFILE temp_file2;
base::trace_event::ProcessMemoryDump pmd_2(nullptr /* session_state */);
base::ScopedFILE temp_file2;
CreateAndSetSmapsFileForTesting(kTestSmaps2, temp_file2);
ProcessMemoryMapsDumpProvider::proc_smaps_for_testing = temp_file2.get();
ProcessMetricsMemoryDumpProvider::proc_smaps_for_testing = temp_file2.get();
pmmdp->OnMemoryDump(dump_args, &pmd_2);
ASSERT_TRUE(pmd_2.has_process_mmaps());
const auto& regions_2 = pmd_2.process_mmaps()->vm_regions();
......@@ -185,6 +227,6 @@ TEST(ProcessMemoryMapsDumpProviderTest, ParseProcSmaps) {
EXPECT_EQ(4 * 1024UL, regions_2[0].byte_stats_private_dirty_resident);
EXPECT_EQ(0 * 1024UL, regions_2[0].byte_stats_swapped);
}
#endif // defined(OS_LINUX) || defined(OS_ANDROID)
} // namespace trace_event
} // namespace base
} // namespace tracing
content/browser/browser_main_loop.cc
View file @ e5fb5cd0
......@@ -32,6 +32,7 @@
#include "base/trace_event/memory_dump_manager.h"
#include "base/trace_event/trace_event.h"
#include "build/build_config.h"
#include "components/tracing/process_metrics_memory_dump_provider.h"
#include "components/tracing/trace_config_file.h"
#include "components/tracing/trace_to_console.h"
#include "components/tracing/tracing_switches.h"
......@@ -696,6 +697,8 @@ void BrowserMainLoop::PostMainMessageLoopStart() {
// Enable memory-infra dump providers.
InitSkiaEventTracer();
tracing::ProcessMetricsMemoryDumpProvider::RegisterForProcess(
base::kNullProcessId);
base::trace_event::MemoryDumpManager::GetInstance()->RegisterDumpProvider(
HostSharedBitmapManager::current(), "HostSharedBitmapManager", nullptr);
base::trace_event::MemoryDumpManager::GetInstance()->RegisterDumpProvider(
......
content/browser/tracing/tracing_controller_impl.cc
View file @ e5fb5cd0
......@@ -12,6 +12,7 @@
#include "base/sys_info.h"
#include "base/trace_event/trace_event.h"
#include "build/build_config.h"
#include "components/tracing/process_metrics_memory_dump_provider.h"
#include "content/browser/tracing/file_tracing_provider_impl.h"
#include "content/browser/tracing/power_tracing_agent.h"
#include "content/browser/tracing/trace_message_filter.h"
......@@ -557,6 +558,13 @@ void TracingControllerImpl::AddTraceMessageFilter(
return;
}
#if defined(OS_LINUX)
// On Linux the browser process dumps process metrics for child process due to
// sandbox.
tracing::ProcessMetricsMemoryDumpProvider::RegisterForProcess(
trace_message_filter->peer_pid());
#endif
trace_message_filters_.insert(trace_message_filter);
if (can_cancel_watch_event()) {
trace_message_filter->SendSetWatchEvent(watch_category_name_,
......@@ -585,6 +593,11 @@ void TracingControllerImpl::RemoveTraceMessageFilter(
return;
}
#if defined(OS_LINUX)
tracing::ProcessMetricsMemoryDumpProvider::UnregisterForProcess(
trace_message_filter->peer_pid());
#endif
// If a filter is removed while a response from that filter is pending then
// simulate the response. Otherwise the response count will be wrong and the
// completion callback will never be executed.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment