Commit 91f12c77 authored by Ramin Halavati's avatar Ramin Halavati Committed by Commit Bot

Reland 'Traffic annotation auditor and unittest updated.'

The following changes were done:
 1- Auditor adds a list of OSes that actively use each annotation to
    annotations.xml, so that removing annotation from one OS and keeping
    it in another would not result in marking it deprecated.
 2- A script is added to tools/traffic_anntotation/scripts, to call all
    scripts that update a give file when annotations.xml is modified, or
    to run unittests that ensure these files are up to date.
 3- 'test-only' switch added to Auditor. When this switch is present,
    auditor only checks if annotations.xml is up to date and issues an
    error it if is not. When switch is not specified, annotations.xml
    may get updated and if so, the above mentioned script is called to
    update downstream files.
 4- Auditor will prevent use of deprecated unique ids.
 5- Unittest checks that annotations.xml is loadable and its content are
    valid.
 6- Unittest calls the script mentioned in (2) to test if downstream
    files are correctly updated.

Bug: 656607
Bug: 690323
Change-Id: I00acbddbe14eeee23317bb9990069e00558a5bb4
Reviewed-on: https://chromium-review.googlesource.com/645088
Commit-Queue: Ramin Halavati <rhalavati@chromium.org>
Reviewed-by: default avatarMartin Šrámek <msramek@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#501206}
Reviewed-on: https://chromium-review.googlesource.com/663724
Cr-Commit-Position: refs/heads/master@{#501584}
parent 31036811
...@@ -38791,6 +38791,7 @@ Called by update_traffic_annotation_histograms.py.--> ...@@ -38791,6 +38791,7 @@ Called by update_traffic_annotation_histograms.py.-->
<int value="8561691" label="safe_browsing_v4_get_hash"/> <int value="8561691" label="safe_browsing_v4_get_hash"/>
<int value="9459438" label="external_policy_fetcher"/> <int value="9459438" label="external_policy_fetcher"/>
<int value="13211343" label="suggestions_image_manager"/> <int value="13211343" label="suggestions_image_manager"/>
<int value="13694792" label="desktop_ios_promotion"/>
<int value="14257952" label="supervised_user_url_filter"/> <int value="14257952" label="supervised_user_url_filter"/>
<int value="15418154" label="ntp_snippets_fetch"/> <int value="15418154" label="ntp_snippets_fetch"/>
<int value="16469669" label="background_fetch_context"/> <int value="16469669" label="background_fetch_context"/>
...@@ -38854,6 +38855,7 @@ Called by update_traffic_annotation_histograms.py.--> ...@@ -38854,6 +38855,7 @@ Called by update_traffic_annotation_histograms.py.-->
<int value="56275203" label="save_file_manager"/> <int value="56275203" label="save_file_manager"/>
<int value="56351037" label="download_web_contents_frame"/> <int value="56351037" label="download_web_contents_frame"/>
<int value="56434025" label="gcm_subscription"/> <int value="56434025" label="gcm_subscription"/>
<int value="56866367" label="pdf_plugin_placeholder"/>
<int value="57144960" label="sync_http_bridge"/> <int value="57144960" label="sync_http_bridge"/>
<int value="57276415" label="chrome_expect_ct_reporter"/> <int value="57276415" label="chrome_expect_ct_reporter"/>
<int value="58963098" label="resource_dispather_host"/> <int value="58963098" label="resource_dispather_host"/>
...@@ -38874,6 +38876,7 @@ Called by update_traffic_annotation_histograms.py.--> ...@@ -38874,6 +38876,7 @@ Called by update_traffic_annotation_histograms.py.-->
<int value="68561428" label="data_reduction_proxy_pingback"/> <int value="68561428" label="data_reduction_proxy_pingback"/>
<int value="68745894" label="safe_browsing_get_full_hash"/> <int value="68745894" label="safe_browsing_get_full_hash"/>
<int value="70126372" label="notification_image_reporter"/> <int value="70126372" label="notification_image_reporter"/>
<int value="70796791" label="content_resource_fetcher"/>
<int value="71251498" label="privet_http_impl"/> <int value="71251498" label="privet_http_impl"/>
<int value="71578042" label="cloud_print_backend"/> <int value="71578042" label="cloud_print_backend"/>
<int value="72354423" label="invalidation_service"/> <int value="72354423" label="invalidation_service"/>
...@@ -38900,6 +38903,7 @@ Called by update_traffic_annotation_histograms.py.--> ...@@ -38900,6 +38903,7 @@ Called by update_traffic_annotation_histograms.py.-->
<int value="88863520" label="autofill_query"/> <int value="88863520" label="autofill_query"/>
<int value="89771989" label="asset_links"/> <int value="89771989" label="asset_links"/>
<int value="91597383" label="gaia_auth_login"/> <int value="91597383" label="gaia_auth_login"/>
<int value="93232258" label="mojo_context_state"/>
<int value="95588446" label="payments_sync_cards"/> <int value="95588446" label="payments_sync_cards"/>
<int value="95682324" label="ranker_url_fetcher"/> <int value="95682324" label="ranker_url_fetcher"/>
<int value="95711309" label="ntp_contextual_suggestions_fetch"/> <int value="95711309" label="ntp_contextual_suggestions_fetch"/>
...@@ -38908,6 +38912,7 @@ Called by update_traffic_annotation_histograms.py.--> ...@@ -38908,6 +38912,7 @@ Called by update_traffic_annotation_histograms.py.-->
<int value="97199008" label="doodle_fetcher"/> <int value="97199008" label="doodle_fetcher"/>
<int value="97978464" label="gaia_auth_fetch_for_uber"/> <int value="97978464" label="gaia_auth_fetch_for_uber"/>
<int value="98123737" label="devtools_interceptor"/> <int value="98123737" label="devtools_interceptor"/>
<int value="99279418" label="rlz_ping"/>
<int value="100545943" label="webui_content_scripts_download"/> <int value="100545943" label="webui_content_scripts_download"/>
<int value="101725581" label="cros_recovery_image_download"/> <int value="101725581" label="cros_recovery_image_download"/>
<int value="101845102" label="blink_resource_loader"/> <int value="101845102" label="blink_resource_loader"/>
...@@ -86,6 +86,13 @@ std::string AuditorResult::ToText() const { ...@@ -86,6 +86,13 @@ std::string AuditorResult::ToText() const {
"word and should be changed.", "word and should be changed.",
details_[0].c_str(), file_path_.c_str(), line_); details_[0].c_str(), file_path_.c_str(), line_);
case AuditorResult::Type::ERROR_DEPRECATED_UNIQUE_ID_HASH_CODE:
DCHECK(details_.size());
return base::StringPrintf(
"Unique id '%s' in '%s:%i' has a hash code similar to a deprecated "
"unique id and should be changed.",
details_[0].c_str(), file_path_.c_str(), line_);
case AuditorResult::Type::ERROR_DUPLICATE_UNIQUE_ID_HASH_CODE: case AuditorResult::Type::ERROR_DUPLICATE_UNIQUE_ID_HASH_CODE:
DCHECK_EQ(details_.size(), 2u); DCHECK_EQ(details_.size(), 2u);
return base::StringPrintf( return base::StringPrintf(
......
...@@ -18,13 +18,15 @@ class AuditorResult { ...@@ -18,13 +18,15 @@ class AuditorResult {
// MISSING_TRAFFIC_ANNOTATION tag. // MISSING_TRAFFIC_ANNOTATION tag.
ERROR_NO_ANNOTATION, // A function is called with NO_ANNOTATION tag. ERROR_NO_ANNOTATION, // A function is called with NO_ANNOTATION tag.
ERROR_SYNTAX, // Annotation syntax is not right. ERROR_SYNTAX, // Annotation syntax is not right.
ERROR_RESERVED_UNIQUE_ID_HASH_CODE, // A unique id has a hash code similar ERROR_RESERVED_UNIQUE_ID_HASH_CODE, // A unique id has a hash code similar
// to a reserved word. // to a reserved word.
ERROR_DUPLICATE_UNIQUE_ID_HASH_CODE, // Two unique ids have similar hash ERROR_DEPRECATED_UNIQUE_ID_HASH_CODE, // A unique id has a hash code
// codes. // equal to a deprecated one.
ERROR_UNIQUE_ID_INVALID_CHARACTER, // A unique id contanins a characer ERROR_DUPLICATE_UNIQUE_ID_HASH_CODE, // Two unique ids have equal hash
// which is not alphanumeric or // codes.
// underline. ERROR_UNIQUE_ID_INVALID_CHARACTER, // A unique id contanins a characer
// which is not alphanumeric or
// underline.
ERROR_MISSING_ANNOTATION, // A function that requires annotation is not ERROR_MISSING_ANNOTATION, // A function that requires annotation is not
// annotated. // annotated.
ERROR_MISSING_EXTRA_ID, // Annotation does not have a valid extra id. ERROR_MISSING_EXTRA_ID, // Annotation does not have a valid extra id.
......
...@@ -14,6 +14,7 @@ ...@@ -14,6 +14,7 @@
#include "build/build_config.h" #include "build/build_config.h"
#include "net/traffic_annotation/network_traffic_annotation.h" #include "net/traffic_annotation/network_traffic_annotation.h"
#include "net/traffic_annotation/network_traffic_annotation_test_helper.h" #include "net/traffic_annotation/network_traffic_annotation_test_helper.h"
#include "tools/traffic_annotation/auditor/traffic_annotation_exporter.h"
#include "tools/traffic_annotation/auditor/traffic_annotation_file_filter.h" #include "tools/traffic_annotation/auditor/traffic_annotation_file_filter.h"
namespace { namespace {
...@@ -135,10 +136,8 @@ bool TrafficAnnotationAuditor::RunClangTool( ...@@ -135,10 +136,8 @@ bool TrafficAnnotationAuditor::RunClangTool(
} }
base::CloseFile(options_file); base::CloseFile(options_file);
base::CommandLine cmdline(source_path_.Append(FILE_PATH_LITERAL("tools")) base::CommandLine cmdline(source_path_.Append(
.Append(FILE_PATH_LITERAL("clang")) FILE_PATH_LITERAL("tools/clang/scripts/run_tool.py")));
.Append(FILE_PATH_LITERAL("scripts"))
.Append(FILE_PATH_LITERAL("run_tool.py")));
#if defined(OS_WIN) #if defined(OS_WIN)
cmdline.PrependWrapper(L"python"); cmdline.PrependWrapper(L"python");
...@@ -325,11 +324,18 @@ void TrafficAnnotationAuditor::PurgeAnnotations( ...@@ -325,11 +324,18 @@ void TrafficAnnotationAuditor::PurgeAnnotations(
extracted_annotations_.end()); extracted_annotations_.end());
} }
void TrafficAnnotationAuditor::CheckDuplicateHashes() { bool TrafficAnnotationAuditor::CheckDuplicateHashes() {
const std::map<int, std::string> reserved_ids = GetReservedUniqueIDs(); const std::map<int, std::string> reserved_ids = GetReservedUniqueIDs();
std::map<int, std::vector<AnnotationID>> collisions; std::map<int, std::vector<AnnotationID>> collisions;
std::set<int> to_be_purged; std::set<int> to_be_purged;
std::set<int> deprecated_ids;
// Load deprecated Hashcodes.
if (!TrafficAnnotationExporter(source_path_)
.GetDeprecatedHashCodes(&deprecated_ids)) {
return false;
}
for (AnnotationInstance& instance : extracted_annotations_) { for (AnnotationInstance& instance : extracted_annotations_) {
// Check if partial and branched completing annotation have an extra id // Check if partial and branched completing annotation have an extra id
...@@ -382,6 +388,16 @@ void TrafficAnnotationAuditor::CheckDuplicateHashes() { ...@@ -382,6 +388,16 @@ void TrafficAnnotationAuditor::CheckDuplicateHashes() {
continue; continue;
} }
// If the id's hash code was formerly used by a deprecated annotation,
// add an error.
if (base::ContainsKey(deprecated_ids, current.hash)) {
errors_.push_back(AuditorResult(
AuditorResult::Type::ERROR_DEPRECATED_UNIQUE_ID_HASH_CODE,
current.text, instance.proto.source().file(),
instance.proto.source().line()));
continue;
}
// Check for collisions. // Check for collisions.
if (!base::ContainsKey(collisions, current.hash)) { if (!base::ContainsKey(collisions, current.hash)) {
collisions[current.hash] = std::vector<AnnotationID>(); collisions[current.hash] = std::vector<AnnotationID>();
...@@ -425,6 +441,7 @@ void TrafficAnnotationAuditor::CheckDuplicateHashes() { ...@@ -425,6 +441,7 @@ void TrafficAnnotationAuditor::CheckDuplicateHashes() {
} }
PurgeAnnotations(to_be_purged); PurgeAnnotations(to_be_purged);
return true;
} }
void TrafficAnnotationAuditor::CheckUniqueIDsFormat() { void TrafficAnnotationAuditor::CheckUniqueIDsFormat() {
...@@ -608,10 +625,12 @@ void TrafficAnnotationAuditor::CheckAnnotationsContents() { ...@@ -608,10 +625,12 @@ void TrafficAnnotationAuditor::CheckAnnotationsContents() {
new_annotations.end()); new_annotations.end());
} }
void TrafficAnnotationAuditor::RunAllChecks() { bool TrafficAnnotationAuditor::RunAllChecks() {
CheckDuplicateHashes(); if (!CheckDuplicateHashes())
return false;
CheckUniqueIDsFormat(); CheckUniqueIDsFormat();
CheckAnnotationsContents(); CheckAnnotationsContents();
CheckAllRequiredFunctionsAreAnnotated(); CheckAllRequiredFunctionsAreAnnotated();
return true;
} }
\ No newline at end of file
...@@ -66,9 +66,10 @@ class TrafficAnnotationAuditor { ...@@ -66,9 +66,10 @@ class TrafficAnnotationAuditor {
AuditorException::ExceptionType exception_type); AuditorException::ExceptionType exception_type);
// Checks to see if any unique id or extra id or their hash code are // Checks to see if any unique id or extra id or their hash code are
// duplicated. Adds errors to |errors_| and purges annotations with duplicate // duplicated, either in currently existing annotations, or in deprecated
// ids. // ones. Adds errors to |errors_| and purges annotations with duplicate ids.
void CheckDuplicateHashes(); // Returns false if any errors happen while checking.
bool CheckDuplicateHashes();
// Checks to see if unique ids only include alphanumeric characters and // Checks to see if unique ids only include alphanumeric characters and
// underline. Adds errors to |errors_| and purges annotations with // underline. Adds errors to |errors_| and purges annotations with
...@@ -89,7 +90,7 @@ class TrafficAnnotationAuditor { ...@@ -89,7 +90,7 @@ class TrafficAnnotationAuditor {
bool CheckIfCallCanBeUnannotated(const CallInstance& call); bool CheckIfCallCanBeUnannotated(const CallInstance& call);
// Performs all checks on extracted annotations and calls. // Performs all checks on extracted annotations and calls.
void RunAllChecks(); bool RunAllChecks();
// Returns a mapping of reserved unique ids' hash codes to the unique ids' // Returns a mapping of reserved unique ids' hash codes to the unique ids'
// texts. This list includes all unique ids that are defined in // texts. This list includes all unique ids that are defined in
......
...@@ -19,7 +19,7 @@ namespace { ...@@ -19,7 +19,7 @@ namespace {
const char* HELP_TEXT = R"( const char* HELP_TEXT = R"(
Traffic Annotation Auditor Traffic Annotation Auditor
Extracts network traffic annotaions from the repository, audits them for errors Extracts network traffic annotaions from the repository, audits them for errors
and coverage, and produces reports. and coverage, produces reports, and updates related files.
Usage: traffic_annotation_auditor [OPTION]... [path_filters] Usage: traffic_annotation_auditor [OPTION]... [path_filters]
...@@ -40,6 +40,12 @@ Options: ...@@ -40,6 +40,12 @@ Options:
--full-run Optional flag asking the tool to run on the whole --full-run Optional flag asking the tool to run on the whole
repository without text filtering files. Using this flag repository without text filtering files. Using this flag
may increase processing time x40. may increase processing time x40.
--test-only Optional flag to request just running tests and not
updating any file. If not specified,
'tools/traffic_annotation/summary/annotations.xml' might
get updated and if it does, 'tools/traffic_annotation/
scripts/annotations_xml_downstream_updater.py will
be called to update downstream files.
--summary-file Optional path to the output file with all annotations. --summary-file Optional path to the output file with all annotations.
--annotations-file Optional path to a TSV output file with all annotations. --annotations-file Optional path to a TSV output file with all annotations.
path_filters Optional paths to filter what files the tool is run on. path_filters Optional paths to filter what files the tool is run on.
...@@ -48,11 +54,32 @@ Example: ...@@ -48,11 +54,32 @@ Example:
traffic_annotation_auditor --build-dir=out/Debug summary-file=report.txt traffic_annotation_auditor --build-dir=out/Debug summary-file=report.txt
)"; )";
const base::FilePath kAnnotationsXmlPath( const base::FilePath kDownstreamUpdater(FILE_PATH_LITERAL(
FILE_PATH_LITERAL("tools/traffic_annotation/summary/annotations.xml")); "tools/traffic_annotation/scripts/annotations_xml_downstream_caller.py"));
} // namespace } // namespace
// Calls |kDownstreamUpdater| script to update files that depend on
// annotations.xml.
bool RunAnnotationDownstreamUpdater(const base::FilePath& source_path) {
base::CommandLine cmdline(source_path.Append(kDownstreamUpdater));
int exit_code;
#if defined(OS_WIN)
cmdline.PrependWrapper(L"python");
exit_code =
system(base::UTF16ToASCII(cmdline.GetCommandLineString()).c_str());
#else
exit_code = system(cmdline.GetCommandLineString().c_str());
#endif
if (exit_code) {
LOG(ERROR) << "Running " << kDownstreamUpdater.MaybeAsASCII()
<< " failed with exit code: " << exit_code;
return false;
}
return true;
}
// Writes a summary of annotations, calls, and errors. // Writes a summary of annotations, calls, and errors.
bool WriteSummaryFile(const base::FilePath& filepath, bool WriteSummaryFile(const base::FilePath& filepath,
const std::vector<AnnotationInstance>& annotations, const std::vector<AnnotationInstance>& annotations,
...@@ -262,6 +289,7 @@ int main(int argc, char* argv[]) { ...@@ -262,6 +289,7 @@ int main(int argc, char* argv[]) {
base::FilePath extractor_input = base::FilePath extractor_input =
command_line.GetSwitchValuePath("extractor-input"); command_line.GetSwitchValuePath("extractor-input");
bool full_run = command_line.HasSwitch("full-run"); bool full_run = command_line.HasSwitch("full-run");
bool test_only = command_line.HasSwitch("test-only");
base::FilePath summary_file = command_line.GetSwitchValuePath("summary-file"); base::FilePath summary_file = command_line.GetSwitchValuePath("summary-file");
base::FilePath annotations_file = base::FilePath annotations_file =
command_line.GetSwitchValuePath("annotations-file"); command_line.GetSwitchValuePath("annotations-file");
...@@ -321,7 +349,10 @@ int main(int argc, char* argv[]) { ...@@ -321,7 +349,10 @@ int main(int argc, char* argv[]) {
return 1; return 1;
// Perform checks. // Perform checks.
auditor.RunAllChecks(); if (!auditor.RunAllChecks()) {
LOG(ERROR) << "Running checks failed.";
return 1;
}
// Write the summary file. // Write the summary file.
if (!summary_file.empty() && if (!summary_file.empty() &&
...@@ -331,15 +362,6 @@ int main(int argc, char* argv[]) { ...@@ -331,15 +362,6 @@ int main(int argc, char* argv[]) {
return 1; return 1;
} }
// Update annotations list.
if (!TrafficAnnotationExporter().UpdateAnnotationsXML(
source_path.Append(kAnnotationsXmlPath),
auditor.extracted_annotations(),
TrafficAnnotationAuditor::GetReservedUniqueIDs())) {
LOG(ERROR) << "Could not update annotations XML.";
return 1;
}
// Write annotations TSV file. // Write annotations TSV file.
if (!annotations_file.empty() && if (!annotations_file.empty() &&
!WriteAnnotationsFile(annotations_file, !WriteAnnotationsFile(annotations_file,
...@@ -348,6 +370,23 @@ int main(int argc, char* argv[]) { ...@@ -348,6 +370,23 @@ int main(int argc, char* argv[]) {
return 1; return 1;
} }
// Test/Update annotations.xml.
TrafficAnnotationExporter exporter(source_path);
if (!exporter.UpdateAnnotations(
auditor.extracted_annotations(),
TrafficAnnotationAuditor::GetReservedUniqueIDs())) {
return 1;
}
if (exporter.modified()) {
if (test_only) {
printf("Error: annotation.xml needs update.\n");
} else if (!exporter.SaveAnnotationsXML() ||
!RunAnnotationDownstreamUpdater(source_path)) {
LOG(ERROR) << "Could not update annotations XML or downstream files.";
return 1;
}
}
// Dump Errors and Warnings to stdout. // Dump Errors and Warnings to stdout.
const std::vector<AuditorResult>& errors = auditor.errors(); const std::vector<AuditorResult>& errors = auditor.errors();
for (const auto& error : errors) { for (const auto& error : errors) {
......
...@@ -11,9 +11,12 @@ ...@@ -11,9 +11,12 @@
#include "base/strings/string_split.h" #include "base/strings/string_split.h"
#include "base/strings/string_util.h" #include "base/strings/string_util.h"
#include "base/strings/stringprintf.h" #include "base/strings/stringprintf.h"
#include "base/strings/utf_string_conversions.h"
#include "build/build_config.h"
#include "net/traffic_annotation/network_traffic_annotation.h" #include "net/traffic_annotation/network_traffic_annotation.h"
#include "net/traffic_annotation/network_traffic_annotation_test_helper.h" #include "net/traffic_annotation/network_traffic_annotation_test_helper.h"
#include "testing/gtest/include/gtest/gtest.h" #include "testing/gtest/include/gtest/gtest.h"
#include "tools/traffic_annotation/auditor/traffic_annotation_exporter.h"
#include "tools/traffic_annotation/auditor/traffic_annotation_file_filter.h" #include "tools/traffic_annotation/auditor/traffic_annotation_file_filter.h"
namespace { namespace {
...@@ -31,7 +34,12 @@ const char* kIrrelevantFiles[] = { ...@@ -31,7 +34,12 @@ const char* kIrrelevantFiles[] = {
const char* kRelevantFiles[] = { const char* kRelevantFiles[] = {
"tools/traffic_annotation/auditor/tests/relevant_file_name_and_content.cc", "tools/traffic_annotation/auditor/tests/relevant_file_name_and_content.cc",
"tools/traffic_annotation/auditor/tests/relevant_file_name_and_content.mm"}; "tools/traffic_annotation/auditor/tests/relevant_file_name_and_content.mm"};
}
const base::FilePath kDownstreamUnittests(
FILE_PATH_LITERAL("tools/traffic_annotation/scripts/"
"annotations_xml_downstream_caller.py"));
} // namespace
using namespace testing; using namespace testing;
...@@ -818,4 +826,28 @@ TEST_F(TrafficAnnotationAuditorTest, CreateCompleteAnnotation) { ...@@ -818,4 +826,28 @@ TEST_F(TrafficAnnotationAuditorTest, CreateCompleteAnnotation) {
NetworkTrafficAnnotation_TrafficSemantics_Destination_LOCAL); NetworkTrafficAnnotation_TrafficSemantics_Destination_LOCAL);
EXPECT_NE(instance.CreateCompleteAnnotation(other, &combination).type(), EXPECT_NE(instance.CreateCompleteAnnotation(other, &combination).type(),
AuditorResult::Type::RESULT_OK); AuditorResult::Type::RESULT_OK);
}
// Tests if Annotations.xml has proper content.
TEST_F(TrafficAnnotationAuditorTest, AnnotationsXML) {
TrafficAnnotationExporter exporter(source_path());
EXPECT_TRUE(exporter.LoadAnnotationsXML());
EXPECT_TRUE(exporter.CheckReportItems());
}
// Tests if downstream files depending on of Annotations.xml are updated.
TEST_F(TrafficAnnotationAuditorTest, AnnotationsDownstreamUnittests) {
base::CommandLine cmdline(source_path().Append(kDownstreamUnittests));
cmdline.AppendSwitch("test");
int tests_result;
#if defined(OS_WIN)
cmdline.PrependWrapper(L"python");
tests_result =
system(base::UTF16ToASCII(cmdline.GetCommandLineString()).c_str());
#else
tests_result = system(cmdline.GetCommandLineString().c_str());
#endif
EXPECT_EQ(0, tests_result);
} }
\ No newline at end of file
...@@ -6,48 +6,54 @@ ...@@ -6,48 +6,54 @@
#define TOOLS_TRAFFIC_ANNOTATION_AUDITOR_TRAFFIC_ANNOTATION_EXPORTER_H_ #define TOOLS_TRAFFIC_ANNOTATION_AUDITOR_TRAFFIC_ANNOTATION_EXPORTER_H_
#include <map> #include <map>
#include <set>
#include <vector> #include <vector>
#include "base/files/file_path.h" #include "base/files/file_path.h"
#include "base/strings/string_util.h"
#include "tools/traffic_annotation/auditor/instance.h" #include "tools/traffic_annotation/auditor/instance.h"
class TrafficAnnotationExporter { class TrafficAnnotationExporter {
public: public:
TrafficAnnotationExporter() = default; TrafficAnnotationExporter(const base::FilePath& source_path);
~TrafficAnnotationExporter() = default; ~TrafficAnnotationExporter();
TrafficAnnotationExporter(const TrafficAnnotationExporter&) = delete; TrafficAnnotationExporter(const TrafficAnnotationExporter&) = delete;
TrafficAnnotationExporter(TrafficAnnotationExporter&&) = delete; TrafficAnnotationExporter(TrafficAnnotationExporter&&) = delete;
// Updates the xml file including annotations unique id, hash code, content // Loads annotations from annotations.xml file into |report_items_|.
// hash code, and a flag specifying that annotation is depricated. bool LoadAnnotationsXML();
bool UpdateAnnotationsXML(const base::FilePath& filepath,
const std::vector<AnnotationInstance>& annotations, // Updates |report_items_| with current set of extracted annotations and
const std::map<int, std::string>& reserved_ids); // reserved ids. Sets the |modified_| flag if any item is updated.
bool UpdateAnnotations(const std::vector<AnnotationInstance>& annotations,
const std::map<int, std::string>& reserved_ids);
// Saves |report_items_| into annotations.xml.
bool SaveAnnotationsXML();
// Produces the list of deprecated hash codes. Returns false if
// annotations.xml is not and cannot be loaded.
bool GetDeprecatedHashCodes(std::set<int>* hash_codes);
bool modified() { return modified_; }
// Runs tests on content of |report_items_|.
bool CheckReportItems();
private: private:
struct ReportItem { struct ReportItem {
ReportItem(std::string id, int hash_code, int content_hash) ReportItem();
: unique_id(id), ReportItem(const ReportItem& other);
unique_id_hash_code(hash_code), ~ReportItem();
deprecation_date(std::string()),
content_hash_code(content_hash) {}
ReportItem(std::string id, int hash_code) : ReportItem(id, hash_code, -1) {}
ReportItem() : ReportItem(std::string(), -1, -1) {}
static bool Compare(const ReportItem& a, const ReportItem& b) {
return base::CompareCaseInsensitiveASCII(a.unique_id, b.unique_id) < 0;
}
std::string unique_id;
int unique_id_hash_code; int unique_id_hash_code;
std::string deprecation_date; std::string deprecation_date;
int content_hash_code; int content_hash_code;
std::vector<std::string> os_list;
}; };
// Loads annotations from the given XML file. std::map<std::string, ReportItem> report_items_;
bool LoadAnnotationsFromXML(const base::FilePath& filepath, const base::FilePath source_path_;
std::vector<ReportItem>* items); bool modified_;
}; };
#endif // TOOLS_TRAFFIC_ANNOTATION_AUDITOR_TRAFFIC_ANNOTATION_EXPORTER_H_ #endif // TOOLS_TRAFFIC_ANNOTATION_AUDITOR_TRAFFIC_ANNOTATION_EXPORTER_H_
\ No newline at end of file
# Traffic Annotation Scripts
This file describes the scripts in `tools/traffic_annotation/scripts`
# annotations_xml_downstream_caller.py
This script calls all scripts that either update a file based on
`annotations.xml`, or test if a file is in sync with it. Call with `test` switch
for test mode.
#!/usr/bin/env python
# Copyright 2017 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
""" Runs all scripts that use
'tools/traffic_annotation/summary/annotations.xml' to update a file, or
test if a file is in sync with it. Use with 'test' switch for test mode.
Add your scripts to PROD_SCRIPTS or TEST_SCRIPTS.
"""
import os.path
import subprocess
import sys
# Add your update scripts here. Each list item will have the script name and the
# list of arguments.
PROD_SCRIPTS = [
["tools/metrics/histograms/update_traffic_annotation_histograms.py", []]]
# Add your test scripts here. Each list item will have the script name and the
# list of arguments.
TEST_SCRIPTS = []
def main(test_mode):
src_path = os.path.abspath(
os.path.join(os.path.dirname(__file__), "..", "..", ".."))
for script in TEST_SCRIPTS if test_mode else PROD_SCRIPTS:
args = [os.path.join(src_path, script[0])]
args += script[1]
if sys.platform == "win32":
args.insert(0, "python")
result = subprocess.call(args)
if result:
if test_mode:
logging.error("Running '%s' script failed with error code: %i." % (
script, result))
return result
return 0
if __name__ == "__main__":
test_mode = (len(sys.argv) > 1 and "test" in sys.argv[1])
sys.exit(main(test_mode))
...@@ -5,16 +5,16 @@ This file describes the `tools/traffic_annotation/summary/annotations.xml`. ...@@ -5,16 +5,16 @@ This file describes the `tools/traffic_annotation/summary/annotations.xml`.
`annotations.xml` includes the summary of all network traffic annotations in `annotations.xml` includes the summary of all network traffic annotations in
Chromium repository. The content includes complete annotations and the merged Chromium repository. The content includes complete annotations and the merged
partial and completing (and branched completing) annotations. partial and completing (and branched completing) annotations.
For each annotation, unique id, hash code of unique id, and hash code of content For each annotation, unique id, hash code of unique id, hash code of the
is presented. If annotation is a reserved one, instead of content hash code, a content, and the list of OSes using this annotation is presented. If annotation
`reserved` attribute is included. is a reserved one, instead of content hash code, a `reserved` attribute is
included.
Once an annotation is removed from the repository, a `deprecated` attribute is Once an annotation is removed from the repository, a `deprecated` attribute is
added to its item in this file, with value equal to the deprecation date. added to its item in this file, with value equal to the deprecation date.
These items can be manually or automatically pruned after sufficient time. These items can be manually or automatically pruned after sufficient time.
# How to Generate/Update. # How to Generate/Update.
Run `traffic_annotation_auditor` to check for annotations correctness and Run `traffic_annotation_auditor` to check for annotations correctness and
automatic update. automatic update. After each modification of`annotations.xml`, auditor calls
After each change in `annotations.xml`, please call `tools/traffic_annotation/scripts/annotations_xml_downstream_updater.py` to
`tools/metrics/histograms/update_traffic_annotation_histograms.py` to update update all users of this file.
annotation enums used in histograms.
\ No newline at end of file
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment